2024-07-24 13:54:36,985 INFO StreamThr :228 [internal.py:wandb_internal():85] W&B internal server running at pid: 228, started at: 2024-07-24 13:54:36.984597 2024-07-24 13:54:36,987 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status 2024-07-24 13:54:37,345 INFO WriterThread:228 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/run-jv4m57pj.wandb 2024-07-24 13:54:37,347 DEBUG SenderThread:228 [sender.py:send():379] send: header 2024-07-24 13:54:37,349 DEBUG SenderThread:228 [sender.py:send():379] send: run 2024-07-24 13:54:37,890 INFO SenderThread:228 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files 2024-07-24 13:54:37,890 INFO SenderThread:228 [sender.py:_start_run_threads():1188] run started: jv4m57pj with start time 1721829276.985577 2024-07-24 13:54:37,900 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: check_version 2024-07-24 13:54:37,900 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: check_version 2024-07-24 13:54:37,975 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: run_start 2024-07-24 13:54:37,987 DEBUG HandlerThread:228 [system_info.py:__init__():26] System info init 2024-07-24 13:54:37,987 DEBUG HandlerThread:228 [system_info.py:__init__():41] System info init done 2024-07-24 13:54:37,988 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 13:54:37,988 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 13:54:37,988 INFO HandlerThread:228 [system_monitor.py:probe():214] Collecting system info 2024-07-24 13:54:37,989 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 13:54:37,989 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 13:54:37,990 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 13:54:37,992 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 13:54:37,993 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 13:54:37,997 DEBUG HandlerThread:228 [system_info.py:probe():152] Probing system 2024-07-24 13:54:37,999 DEBUG HandlerThread:228 [gitlib.py:_init_repo():56] git repository is invalid 2024-07-24 13:54:37,999 DEBUG HandlerThread:228 [system_info.py:probe():200] Probing system done 2024-07-24 13:54:37,999 DEBUG HandlerThread:228 [system_monitor.py:probe():223] {'os': 'Linux-5.15.154+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-07-24T13:54:37.997260', 'startedAt': '2024-07-24T13:54:36.975554', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': '21f06c5830ff', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.128, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.128, 'min': 0.0, 'max': 0.0}, {'current': 2000.128, 'min': 0.0, 'max': 0.0}, {'current': 2000.128, 'min': 0.0, 'max': 0.0}, {'current': 2000.128, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5777.571537017822}}, 'gpu': 'Tesla P100-PCIE-16GB', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla P100-PCIE-16GB', 'memory_total': 17179869184}], 'memory': {'total': 31.357574462890625}} 2024-07-24 13:54:38,000 INFO HandlerThread:228 [system_monitor.py:probe():224] Finished collecting system info 2024-07-24 13:54:38,000 INFO HandlerThread:228 [system_monitor.py:probe():227] Publishing system info 2024-07-24 13:54:38,000 DEBUG HandlerThread:228 [system_info.py:_save_conda():209] Saving list of conda packages installed into the current environment 2024-07-24 13:54:38,892 INFO Thread-12 :228 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/conda-environment.yaml 2024-07-24 13:54:53,014 ERROR HandlerThread:228 [system_info.py:_save_conda():223] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 216, in _save_conda subprocess.call( File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call return p.wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait return self._wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait raise TimeoutExpired(self.args, timeout) subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds 2024-07-24 13:54:53,016 DEBUG HandlerThread:228 [system_info.py:_save_conda():224] Saving conda packages done 2024-07-24 13:54:53,017 INFO HandlerThread:228 [system_monitor.py:probe():229] Finished publishing system info 2024-07-24 13:54:53,025 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:54:53,025 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: keepalive 2024-07-24 13:54:53,025 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:54:53,025 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: keepalive 2024-07-24 13:54:53,025 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:54:53,026 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: keepalive 2024-07-24 13:54:53,026 DEBUG SenderThread:228 [sender.py:send():379] send: files 2024-07-24 13:54:53,026 INFO SenderThread:228 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2024-07-24 13:54:53,895 INFO Thread-12 :228 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/wandb-metadata.json 2024-07-24 13:54:53,981 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: python_packages 2024-07-24 13:54:53,981 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: python_packages 2024-07-24 13:54:53,984 DEBUG SenderThread:228 [sender.py:send():379] send: telemetry 2024-07-24 13:54:53,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:54:53,985 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:54:53,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:54:53,991 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 13:54:53,991 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 13:54:53,991 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 13:54:53,991 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 13:54:53,993 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 13:54:53,994 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 13:54:53,994 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 13:54:53,994 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 13:54:54,000 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 13:54:54,000 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 13:54:54,000 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 13:54:54,272 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 13:54:54,527 INFO wandb-upload_0:228 [upload_job.py:push():130] Uploaded file /tmp/tmpwiz6o3k3wandb/vvo32rpu-wandb-metadata.json 2024-07-24 13:54:54,896 INFO Thread-12 :228 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/requirements.txt 2024-07-24 13:54:54,982 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:54:55,983 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:54:56,983 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:54:57,983 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:54:58,274 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:54:58,983 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:54:59,983 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:00,983 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:01,983 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:02,984 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:03,983 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:55:03,984 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:04,984 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:05,984 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:06,984 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:07,984 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:08,981 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:55:08,982 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:55:09,022 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:09,263 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:55:09,900 INFO Thread-12 :228 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/config.yaml 2024-07-24 13:55:09,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:10,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:11,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:12,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:13,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:14,983 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:55:14,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:15,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:16,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:17,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:18,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:19,983 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:55:19,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:20,867 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 13:55:20,867 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 13:55:20,867 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 13:55:20,867 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 13:55:20,868 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 13:55:20,869 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 13:55:20,870 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 13:55:20,871 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 13:55:20,872 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 13:55:20,908 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 13:55:20,908 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 13:55:20,908 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 13:55:20,908 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 13:55:20,908 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 13:55:20,908 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 13:55:20,909 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 13:55:20,909 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 13:55:20,914 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 13:55:20,914 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 13:55:20,914 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 13:55:20,915 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 13:55:20,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:21,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:22,987 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:23,982 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:55:23,982 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:55:24,023 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:24,987 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:25,148 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 13:55:25,148 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 13:55:25,148 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 13:55:25,149 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 13:55:25,149 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 13:55:25,150 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 13:55:25,152 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 13:55:25,152 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 13:55:25,153 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 13:55:25,189 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 13:55:25,189 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 13:55:25,189 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 13:55:25,189 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 13:55:25,190 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 13:55:25,190 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 13:55:25,190 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 13:55:25,190 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 13:55:25,196 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 13:55:25,196 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 13:55:25,196 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 13:55:25,197 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 13:55:25,197 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:55:25,987 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:26,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:27,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:28,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:29,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:30,984 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:55:30,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:31,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:32,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:33,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:34,989 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:35,989 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:36,984 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:55:36,989 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:37,989 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:38,982 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:55:38,982 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:55:39,023 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:39,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:40,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:41,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:42,259 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:55:42,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:43,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:44,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:45,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:46,991 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:47,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:55:47,991 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:48,991 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:49,991 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:50,991 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:51,991 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:52,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:55:52,992 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:53,982 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:55:53,982 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:55:54,023 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:54,992 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:55,992 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:56,992 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:57,993 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:58,261 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:55:58,993 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:55:59,993 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:00,993 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:01,993 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:02,993 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:03,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:56:03,993 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:04,994 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:05,994 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:06,994 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:07,994 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:08,982 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:56:08,983 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:56:09,023 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:09,232 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:56:09,994 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:10,994 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:11,995 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:12,995 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:13,995 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:14,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:56:14,995 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:15,995 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:16,995 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:17,996 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:18,996 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:19,996 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:20,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:56:20,996 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:21,996 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:22,996 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:23,982 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:56:23,983 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:56:24,023 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:24,997 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:25,997 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:26,291 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:56:26,997 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:27,998 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:28,997 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:29,998 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:30,998 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:31,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:56:31,998 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:32,998 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:33,998 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:34,999 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:35,999 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:36,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:56:36,999 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:37,999 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:38,982 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:56:38,983 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:56:39,023 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:39,999 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:41,000 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:42,000 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:42,257 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:56:43,000 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:44,000 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:45,000 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:46,000 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:47,001 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:47,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:56:48,001 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:49,001 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:50,001 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:51,001 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:52,002 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:53,001 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:53,983 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:56:53,983 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:56:54,023 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:54,254 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:56:55,002 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:56,002 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:57,002 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:58,002 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:59,002 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:56:59,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:57:00,003 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:00,399 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 13:57:00,399 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 13:57:00,399 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 13:57:00,399 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 13:57:00,400 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 13:57:00,400 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 13:57:00,401 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 13:57:00,401 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 13:57:00,402 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 13:57:01,003 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:02,003 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:03,003 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:03,437 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 13:57:03,437 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 13:57:03,437 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 13:57:03,437 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 13:57:03,438 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 13:57:03,438 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 13:57:03,438 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 13:57:03,439 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 13:57:03,446 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 13:57:03,446 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 13:57:03,446 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 13:57:03,446 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 13:57:04,003 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:04,987 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:57:05,003 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:06,004 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:07,004 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:08,004 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:08,983 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:57:08,983 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:57:09,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:10,004 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:10,017 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 13:57:10,017 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 13:57:10,017 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 13:57:10,017 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 13:57:10,018 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 13:57:10,018 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 13:57:10,020 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 13:57:10,021 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 13:57:10,021 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 13:57:10,059 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 13:57:10,059 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 13:57:10,059 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 13:57:10,059 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 13:57:10,059 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 13:57:10,060 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 13:57:10,060 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 13:57:10,061 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 13:57:10,065 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 13:57:10,066 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 13:57:10,066 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 13:57:10,066 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 13:57:10,067 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:57:11,004 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:12,005 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:13,005 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:14,005 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:15,005 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:15,987 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:57:16,005 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:16,808 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 13:57:16,808 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 13:57:16,808 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 13:57:16,808 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 13:57:16,809 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 13:57:16,809 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 13:57:16,811 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 13:57:16,812 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 13:57:16,812 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 13:57:16,849 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 13:57:16,849 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 13:57:16,849 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 13:57:16,849 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 13:57:16,849 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 13:57:16,849 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 13:57:16,850 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 13:57:16,850 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 13:57:16,854 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 13:57:16,855 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 13:57:16,855 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 13:57:16,855 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 13:57:17,005 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:18,006 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:19,006 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:20,006 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:21,006 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:21,987 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:57:22,007 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:23,007 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:23,983 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:57:23,984 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:57:24,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:25,007 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:26,007 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:27,007 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:27,201 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:57:28,008 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:29,008 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:30,008 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:31,008 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:32,008 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:32,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:57:33,009 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:34,009 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:35,009 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:36,009 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:37,009 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:38,010 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:38,135 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 13:57:38,136 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 13:57:38,136 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 13:57:38,136 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 13:57:38,136 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 13:57:38,137 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 13:57:38,139 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 13:57:38,141 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 13:57:38,144 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 13:57:38,198 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 13:57:38,198 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 13:57:38,198 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 13:57:38,198 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 13:57:38,199 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 13:57:38,199 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 13:57:38,199 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 13:57:38,199 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 13:57:38,205 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 13:57:38,205 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 13:57:38,205 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 13:57:38,206 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 13:57:38,206 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:57:38,983 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:57:38,983 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:57:39,009 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:40,010 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:41,010 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:42,010 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:43,010 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:43,231 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:57:44,010 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:45,010 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:46,010 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:47,010 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:48,011 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:48,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:57:49,011 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:50,011 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:51,011 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:52,011 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:53,012 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:53,983 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:57:53,984 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:57:54,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:54,220 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:57:55,012 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:56,012 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:57,012 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:58,013 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:59,013 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:57:59,989 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:58:00,013 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:01,013 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:02,014 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:03,014 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:04,014 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:05,014 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:05,989 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:58:06,014 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:07,015 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:08,015 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:08,983 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:58:08,984 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:58:09,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:10,015 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:11,015 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:11,301 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:58:12,015 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:13,016 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:14,016 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:15,016 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:16,016 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:16,989 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:58:17,017 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:18,017 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:19,017 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:20,017 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:21,017 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:21,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:58:22,018 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:23,018 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:23,984 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:58:23,984 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:58:24,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:25,018 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:26,018 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:27,019 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:27,246 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:58:28,019 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:29,019 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:30,019 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:31,019 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:32,019 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:32,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:58:33,019 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:34,020 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:35,020 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:36,020 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:37,020 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:37,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:58:38,021 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:38,984 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:58:38,984 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:58:39,025 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:40,021 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:41,021 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:42,021 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:43,021 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:43,289 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:58:44,021 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:45,022 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:46,022 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:47,022 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:48,022 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:48,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:58:49,022 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:50,023 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:51,023 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:52,023 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:53,023 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:53,984 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:58:53,984 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:58:54,025 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:54,223 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:58:55,023 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:56,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:57,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:57,160 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 13:58:57,160 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 13:58:57,160 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 13:58:57,160 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 13:58:57,161 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 13:58:57,162 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 13:58:57,163 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 13:58:57,163 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 13:58:57,165 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 13:58:58,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:59,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:58:59,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:59:00,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:01,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:02,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:03,025 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:04,025 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:04,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:59:05,025 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:06,199 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:07,139 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:08,140 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:08,984 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:59:08,984 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:59:09,140 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:10,140 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:10,231 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:59:11,140 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:12,140 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:13,140 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:14,140 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:15,140 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:15,991 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:59:16,141 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:17,141 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:18,141 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:19,141 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:19,552 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 13:59:19,552 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 13:59:19,552 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 13:59:19,552 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 13:59:19,552 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 13:59:19,553 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 13:59:19,554 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 13:59:19,554 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 13:59:19,559 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 13:59:19,560 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 13:59:19,560 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 13:59:19,560 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 13:59:20,141 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:21,142 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:21,561 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:59:22,142 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:23,142 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:23,984 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:59:23,985 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:59:24,142 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:25,142 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:26,142 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:27,143 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:27,222 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:59:28,143 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:29,143 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:30,143 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:31,143 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:32,144 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:32,992 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:59:33,144 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:34,144 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:35,144 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:36,144 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:37,145 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:37,992 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:59:38,145 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:38,984 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:59:38,985 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:59:39,145 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:40,145 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:41,145 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:42,145 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:43,146 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:43,224 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:59:44,146 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:45,146 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:46,146 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:47,146 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:48,147 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:48,992 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:59:49,147 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:50,147 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:51,147 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:51,240 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 13:59:51,240 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 13:59:51,240 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 13:59:51,240 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 13:59:51,240 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 13:59:51,241 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 13:59:51,242 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 13:59:51,242 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 13:59:51,244 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 13:59:52,479 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:53,177 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 13:59:53,177 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 13:59:53,177 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 13:59:53,178 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 13:59:53,178 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 13:59:53,179 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 13:59:53,179 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 13:59:53,179 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 13:59:53,185 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 13:59:53,185 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 13:59:53,186 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 13:59:53,186 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 13:59:53,344 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:53,984 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 13:59:53,985 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 13:59:54,222 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 13:59:54,344 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:55,344 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:56,345 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:57,345 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:58,345 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:59,345 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 13:59:59,992 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:00:00,345 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:01,345 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:02,345 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:03,346 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:04,438 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:05,351 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:05,993 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:00:06,351 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:07,351 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:08,351 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:08,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:00:08,985 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:00:09,351 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:10,352 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:11,266 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:00:11,352 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:11,723 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:00:11,724 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:00:11,724 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:00:11,724 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:00:11,724 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:00:11,725 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:00:11,728 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:00:11,729 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:00:11,730 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:00:12,117 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:00:12,117 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:00:12,117 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:00:12,117 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:00:12,118 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:00:12,118 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:00:12,120 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:00:12,120 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:00:12,126 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:00:12,126 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:00:12,126 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:00:12,127 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:00:12,352 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:13,352 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:14,352 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:15,353 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:16,353 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:16,993 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:00:17,353 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:18,354 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:18,421 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:00:18,421 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:00:18,422 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:00:18,422 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:00:18,422 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:00:18,423 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:00:18,424 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:00:18,424 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:00:18,425 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:00:19,108 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:00:19,108 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:00:19,108 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:00:19,108 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:00:19,108 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:00:19,108 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:00:19,109 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:00:19,110 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:00:19,115 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:00:19,115 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:00:19,115 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:00:19,116 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:00:19,354 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:20,354 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:21,354 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:22,117 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:00:22,355 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:23,355 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:23,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:00:23,985 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:00:24,355 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:25,355 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:26,355 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:27,266 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:00:27,355 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:28,356 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:29,356 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:30,356 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:31,356 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:32,356 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:32,994 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:00:33,357 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:34,357 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:35,357 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:36,357 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:37,357 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:37,994 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:00:38,357 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:38,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:00:38,985 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:00:39,357 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:40,358 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:41,358 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:42,358 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:43,280 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:00:43,358 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:44,359 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:44,819 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:00:44,819 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:00:44,819 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:00:44,820 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:00:44,820 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:00:44,821 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:00:44,824 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:00:44,825 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:00:44,826 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:00:45,359 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:46,359 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:47,360 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:48,360 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:48,994 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:00:49,360 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:50,360 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:51,360 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:52,361 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:53,361 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:53,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:00:53,986 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:00:54,222 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:00:54,361 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:55,361 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:56,361 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:57,362 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:58,361 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:59,361 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:00:59,595 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:00:59,595 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:00:59,595 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:00:59,595 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:00:59,596 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:00:59,596 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:00:59,596 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:00:59,597 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:00:59,603 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:00:59,603 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:00:59,603 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:00:59,604 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:00:59,604 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:01:00,362 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:01,362 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:01,516 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:01:01,516 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:01:01,516 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:01:01,516 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:01:01,517 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:01:01,518 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:01:01,518 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:01:01,519 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:01:01,519 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:01:01,557 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:01:01,557 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:01:01,557 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:01:01,557 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:01:01,557 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:01:01,557 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:01:01,558 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:01:01,559 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:01:01,564 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:01:01,564 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:01:01,564 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:01:01,565 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:01:02,029 INFO Thread-12 :228 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/output.log 2024-07-24 14:01:02,362 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:03,362 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:04,030 INFO Thread-12 :228 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/output.log 2024-07-24 14:01:04,362 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:04,995 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:01:05,362 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:06,363 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:07,363 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:08,363 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:08,985 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:01:08,986 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:01:09,363 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:10,240 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:01:10,363 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:11,363 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:12,363 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:13,364 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:14,364 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:15,364 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:15,995 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:01:16,364 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:17,364 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:18,364 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:19,364 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:20,365 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:20,996 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:01:21,365 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:22,365 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:23,365 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:23,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:01:23,986 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:01:24,365 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:25,366 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:26,255 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:01:26,364 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:01:26,364 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:01:26,364 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:01:26,364 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:01:26,365 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:01:26,366 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:01:26,367 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:01:26,368 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:01:26,369 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:01:26,405 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:26,405 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:01:26,405 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:01:26,405 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:01:26,406 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:01:26,406 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:01:26,406 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:01:26,407 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:01:26,407 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:01:26,412 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:01:26,412 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:01:26,412 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:01:26,413 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:01:27,366 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:28,366 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:28,970 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:01:28,970 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:01:28,970 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:01:28,970 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:01:28,971 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:01:28,972 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:01:28,972 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:01:28,976 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:01:28,978 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:01:29,010 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:01:29,010 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:01:29,010 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:01:29,010 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:01:29,010 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:01:29,011 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:01:29,011 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:01:29,011 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:01:29,016 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:01:29,017 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:01:29,017 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:01:29,017 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:01:29,367 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:30,155 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:01:30,155 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:01:30,156 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:01:30,156 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:01:30,156 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:01:30,157 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:01:30,158 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:01:30,158 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:01:30,159 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:01:30,209 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:01:30,209 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:01:30,210 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:01:30,210 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:01:30,210 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:01:30,210 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:01:30,211 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:01:30,211 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:01:30,218 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:01:30,218 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:01:30,218 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:01:30,219 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:01:30,367 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:31,367 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:32,220 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:01:32,367 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:33,368 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:34,368 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:35,368 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:36,368 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:37,368 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:37,996 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:01:38,043 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:01:38,043 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:01:38,043 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:01:38,043 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:01:38,044 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:01:38,045 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:01:38,046 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:01:38,047 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:01:38,049 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:01:38,085 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:01:38,085 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:01:38,085 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:01:38,085 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:01:38,085 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:01:38,086 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:01:38,086 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:01:38,086 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:01:38,091 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:01:38,092 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:01:38,092 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:01:38,092 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:01:38,368 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:38,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:01:38,986 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:01:39,368 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:39,473 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:01:39,474 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:01:39,474 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:01:39,474 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:01:39,474 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:01:39,475 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:01:39,475 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:01:39,479 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:01:39,479 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:01:40,369 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:41,369 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:42,369 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:43,281 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:01:43,369 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:44,370 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:45,370 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:46,370 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:47,370 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:48,371 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:48,996 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:01:49,371 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:50,371 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:51,371 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:52,371 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:53,371 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:53,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:01:53,987 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:01:54,177 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:01:54,177 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:01:54,177 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:01:54,177 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:01:54,178 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:01:54,178 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:01:54,178 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:01:54,179 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:01:54,186 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:01:54,186 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:01:54,187 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:01:54,187 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:01:54,187 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:01:54,187 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:01:54,188 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:01:54,189 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:01:54,189 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:01:54,189 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:01:54,190 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:01:54,190 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:01:54,190 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:01:54,190 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:01:54,191 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:01:54,207 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:01:54,207 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:01:54,207 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:01:54,371 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:55,371 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:56,049 INFO Thread-12 :228 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/output.log 2024-07-24 14:01:56,372 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:57,372 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:58,372 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:59,372 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:01:59,996 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:02:00,372 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:01,372 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:02,373 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:03,373 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:04,373 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:05,373 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:05,997 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:02:06,373 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:07,374 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:08,374 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:08,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:02:08,986 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:02:09,374 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:10,374 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:11,246 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:02:11,374 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:12,375 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:13,375 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:14,375 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:15,375 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:16,375 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:16,727 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:02:16,727 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:02:16,727 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:02:16,727 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:02:16,727 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:02:16,728 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:02:16,729 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:02:16,730 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:02:16,730 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:02:16,997 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:02:17,375 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:18,107 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:02:18,107 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:02:18,107 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:02:18,107 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:02:18,108 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:02:18,108 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:02:18,108 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:02:18,109 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:02:18,115 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:02:18,115 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:02:18,116 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:02:18,116 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:02:18,376 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:19,376 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:19,444 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:02:19,444 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:02:19,444 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:02:19,444 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:02:19,445 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:02:19,445 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:02:19,448 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:02:19,451 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:02:19,451 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:02:19,485 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:02:19,485 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:02:19,485 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:02:19,485 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:02:19,485 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:02:19,485 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:02:19,486 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:02:19,486 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:02:19,491 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:02:19,491 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:02:19,491 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:02:19,492 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:02:20,376 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:21,376 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:22,377 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:22,493 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:02:23,377 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:23,512 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:02:23,512 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:02:23,512 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:02:23,512 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:02:23,513 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:02:23,514 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:02:23,514 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:02:23,516 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:02:23,518 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:02:23,553 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:02:23,553 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:02:23,553 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:02:23,553 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:02:23,553 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:02:23,553 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:02:23,554 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:02:23,554 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:02:23,559 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:02:23,559 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:02:23,559 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:02:23,560 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:02:23,978 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:02:23,978 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:02:23,978 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:02:23,978 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:02:23,978 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:02:23,979 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:02:23,980 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:02:23,982 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:02:23,983 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:02:23,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:02:23,986 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:02:23,991 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:02:23,991 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:02:23,991 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:02:23,991 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:02:23,992 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:02:23,992 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:02:23,992 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:02:23,993 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:02:23,999 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:02:23,999 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:02:23,999 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:02:24,234 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:02:24,377 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:25,377 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:26,377 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:27,377 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:28,236 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:02:28,377 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:29,378 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:29,912 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:02:29,912 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:02:29,912 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:02:29,912 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:02:29,913 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:02:29,914 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:02:29,916 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:02:29,916 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:02:29,917 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:02:29,953 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:02:29,953 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:02:29,953 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:02:29,953 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:02:29,954 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:02:29,954 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:02:29,954 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:02:29,954 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:02:29,959 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:02:29,959 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:02:29,959 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:02:29,960 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:02:30,378 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:31,131 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:02:31,132 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:02:31,132 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:02:31,132 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:02:31,132 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:02:31,133 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:02:31,134 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:02:31,135 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:02:31,136 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:02:31,378 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:32,378 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:33,378 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:33,961 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:02:34,379 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:35,379 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:36,379 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:37,379 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:37,464 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:02:37,464 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:02:37,464 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:02:37,464 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:02:37,464 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:02:37,464 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:02:37,465 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:02:37,466 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:02:37,472 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:02:37,472 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:02:37,472 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:02:37,472 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:02:38,379 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:38,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:02:38,987 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:02:39,216 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:02:39,379 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:40,379 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:41,380 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:42,380 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:43,380 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:43,841 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:02:43,841 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:02:43,841 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:02:43,842 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:02:43,842 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:02:43,843 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:02:43,844 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:02:43,848 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:02:43,849 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:02:43,882 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:02:43,882 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:02:43,882 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:02:43,883 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:02:43,883 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:02:43,884 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:02:43,884 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:02:43,884 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:02:43,889 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:02:43,890 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:02:43,890 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:02:43,890 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:02:44,380 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:44,998 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:02:45,380 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:46,066 INFO Thread-12 :228 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/output.log 2024-07-24 14:02:46,380 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:47,380 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:48,381 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:49,381 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:49,998 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:02:50,381 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:51,381 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:52,381 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:53,382 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:53,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:02:53,987 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:02:54,382 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:55,254 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:02:55,382 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:56,382 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:57,115 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:02:57,115 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:02:57,115 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:02:57,116 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:02:57,116 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:02:57,119 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:02:57,119 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:02:57,120 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:02:57,121 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:02:57,382 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:58,383 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:02:58,459 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:02:58,460 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:02:58,460 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:02:58,461 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:02:58,461 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:02:58,461 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:02:58,461 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:02:58,462 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:02:58,468 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:02:58,469 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:02:58,469 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:02:58,469 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:02:59,383 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:00,383 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:00,874 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:03:00,875 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:03:00,875 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:03:00,875 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:03:00,875 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:03:00,876 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:03:00,877 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:03:00,878 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:03:00,878 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:03:00,915 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:03:00,915 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:03:00,915 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:03:00,915 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:03:00,915 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:03:00,915 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:03:00,916 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:03:00,916 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:03:00,921 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:03:00,921 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:03:00,921 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:03:00,922 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:03:00,922 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:03:01,160 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:03:01,160 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:03:01,160 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:03:01,160 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:03:01,161 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:03:01,162 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:03:01,162 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:03:01,163 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:03:01,163 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:03:01,163 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:03:01,164 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:03:01,164 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:03:01,164 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:03:01,164 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:03:01,166 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:03:01,168 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:03:01,174 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:03:01,175 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:03:01,175 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:03:01,383 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:01,638 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:03:01,638 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:03:01,638 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:03:01,638 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:03:01,639 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:03:01,641 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:03:01,642 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:03:01,642 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:03:01,643 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:03:01,679 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:03:01,679 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:03:01,679 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:03:01,679 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:03:01,679 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:03:01,679 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:03:01,680 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:03:01,680 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:03:01,685 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:03:01,686 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:03:01,686 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:03:01,686 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:03:02,383 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:03,383 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:04,384 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:05,384 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:05,999 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:03:06,122 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:03:06,122 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:03:06,122 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:03:06,123 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:03:06,123 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:03:06,124 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:03:06,125 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:03:06,127 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:03:06,128 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:03:06,163 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:03:06,163 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:03:06,163 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:03:06,163 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:03:06,163 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:03:06,163 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:03:06,164 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:03:06,164 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:03:06,169 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:03:06,169 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:03:06,170 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:03:06,170 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:03:06,384 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:07,384 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:07,460 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:03:07,460 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:03:07,460 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:03:07,460 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:03:07,461 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:03:07,461 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:03:07,462 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:03:07,464 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:03:07,465 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:03:08,385 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:08,987 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:03:08,987 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:03:09,385 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:10,385 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:11,271 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:03:11,385 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:12,385 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:13,386 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:14,385 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:15,386 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:16,386 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:16,999 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:03:17,387 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:18,387 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:19,387 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:20,387 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:21,387 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:22,000 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:03:22,268 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:03:22,269 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:03:22,269 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:03:22,269 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:03:22,269 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:03:22,270 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:03:22,270 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:03:22,270 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:03:22,277 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:03:22,277 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:03:22,277 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:03:22,278 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:03:22,309 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:03:22,309 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:03:22,309 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:03:22,309 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:03:22,309 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:03:22,310 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:03:22,310 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:03:22,310 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:03:22,311 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:03:22,312 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:03:22,312 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:03:22,313 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:03:22,313 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:03:22,314 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:03:22,314 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:03:22,315 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:03:22,327 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:03:22,327 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:03:22,328 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:03:22,387 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:23,387 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:23,987 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:03:23,987 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:03:24,079 INFO Thread-12 :228 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/output.log 2024-07-24 14:03:24,387 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:25,387 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:26,387 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:27,272 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:03:27,387 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:28,388 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:29,388 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:30,388 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:31,388 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:32,388 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:33,000 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:03:33,388 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:34,388 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:35,389 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:36,389 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:37,389 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:38,000 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:03:38,389 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:38,987 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:03:38,987 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:03:39,389 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:40,389 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:41,390 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:42,390 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:43,256 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:03:43,390 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:44,390 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:45,390 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:46,391 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:47,100 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:03:47,101 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:03:47,101 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:03:47,101 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:03:47,101 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:03:47,102 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:03:47,104 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:03:47,105 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:03:47,106 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:03:47,391 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:48,384 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:03:48,384 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:03:48,384 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:03:48,384 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:03:48,385 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:03:48,385 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:03:48,385 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:03:48,386 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:03:48,392 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:03:48,392 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:03:48,392 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:03:48,393 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:03:48,393 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:03:48,425 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:49,391 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:50,391 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:51,391 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:52,392 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:53,392 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:53,987 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:03:53,988 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:03:54,221 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:03:54,392 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:55,392 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:56,392 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:57,392 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:58,393 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:03:59,393 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:00,000 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:04:00,393 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:00,450 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:04:00,450 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:04:00,450 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:04:00,450 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:04:00,451 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:04:00,451 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:04:00,454 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:04:00,455 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:04:00,456 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:04:00,490 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:04:00,490 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:04:00,490 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:04:00,490 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:04:00,490 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:04:00,491 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:04:00,491 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:04:00,492 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:04:00,497 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:04:00,497 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:04:00,498 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:04:00,498 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:04:01,393 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:01,778 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:04:01,778 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:04:01,778 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:04:01,778 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:04:01,779 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:04:01,780 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:04:01,781 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:04:01,781 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:04:01,783 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:04:01,819 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:04:01,819 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:04:01,819 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:04:01,819 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:04:01,819 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:04:01,819 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:04:01,820 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:04:01,820 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:04:01,825 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:04:01,825 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:04:01,825 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:04:01,826 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:04:02,393 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:02,848 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:04:02,848 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:04:02,848 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:04:02,848 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:04:02,849 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:04:02,850 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:04:02,851 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:04:02,852 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:04:02,854 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:04:02,889 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:04:02,889 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:04:02,889 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:04:02,889 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:04:02,889 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:04:02,889 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:04:02,890 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:04:02,890 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:04:02,895 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:04:02,895 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:04:02,895 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:04:02,896 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:04:03,393 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:04,394 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:05,000 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:04:05,394 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:06,394 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:06,753 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:04:06,754 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:04:06,754 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:04:06,754 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:04:06,755 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:04:06,755 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:04:06,757 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:04:06,760 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:04:06,760 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:04:06,794 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:04:06,794 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:04:06,794 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:04:06,794 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:04:06,794 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:04:06,794 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:04:06,795 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:04:06,795 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:04:06,800 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:04:06,800 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:04:06,800 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:04:06,801 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:04:07,394 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:08,149 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:04:08,150 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:04:08,150 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:04:08,150 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:04:08,150 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:04:08,151 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:04:08,152 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:04:08,153 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:04:08,154 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:04:08,394 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:08,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:04:08,989 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:04:09,395 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:10,230 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:04:10,395 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:11,395 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:11,760 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:04:11,760 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:04:11,760 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:04:11,761 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:04:11,761 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:04:11,761 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:04:11,762 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:04:11,762 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:04:11,768 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:04:11,768 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:04:11,768 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:04:11,768 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:04:12,395 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:13,395 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:14,395 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:15,395 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:16,001 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:04:16,395 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:17,396 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:18,202 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:04:18,202 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:04:18,202 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:04:18,203 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:04:18,203 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:04:18,204 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:04:18,209 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:04:18,209 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:04:18,209 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:04:18,244 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:04:18,244 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:04:18,244 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:04:18,244 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:04:18,244 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:04:18,245 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:04:18,245 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:04:18,245 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:04:18,251 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:04:18,251 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:04:18,251 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:04:18,251 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:04:18,395 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:19,396 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:20,099 INFO Thread-12 :228 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/output.log 2024-07-24 14:04:20,396 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:21,001 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:04:21,396 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:22,396 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:23,396 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:23,987 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:04:23,988 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:04:24,396 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:25,397 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:26,193 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:04:26,397 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:27,397 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:28,397 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:29,397 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:30,397 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:31,398 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:32,001 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:04:32,398 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:33,398 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:34,398 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:35,398 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:36,398 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:37,398 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:38,001 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:04:38,399 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:38,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:04:38,988 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:04:39,399 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:40,399 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:41,399 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:42,399 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:43,256 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:04:43,399 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:44,400 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:45,400 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:46,400 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:47,400 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:48,400 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:49,001 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:04:49,400 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:50,400 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:51,401 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:52,401 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:53,401 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:53,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:04:53,988 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:04:54,215 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:04:54,401 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:55,401 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:55,882 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:04:55,882 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:04:55,882 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:04:55,883 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:04:55,883 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:04:55,884 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:04:55,888 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:04:55,891 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:04:55,892 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:04:55,924 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:04:55,924 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:04:55,924 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:04:55,924 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:04:55,924 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:04:55,924 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:04:55,925 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:04:55,925 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:04:55,931 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:04:55,931 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:04:55,931 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:04:55,932 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:04:56,166 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:04:56,166 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:04:56,166 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:04:56,167 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:04:56,167 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:04:56,168 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:04:56,168 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:04:56,168 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:04:56,168 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:04:56,169 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:04:56,169 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:04:56,169 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:04:56,171 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:04:56,172 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:04:56,172 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:04:56,402 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:56,601 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:04:56,602 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:04:56,602 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:04:56,602 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:04:56,602 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:04:56,603 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:04:56,603 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:04:56,604 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:04:56,606 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:04:56,642 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:04:56,642 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:04:56,642 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:04:56,642 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:04:56,642 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:04:56,642 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:04:56,643 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:04:56,643 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:04:56,648 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:04:56,648 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:04:56,648 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:04:56,649 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:04:57,402 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:58,402 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:04:59,402 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:00,002 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:05:00,116 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:05:00,116 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:05:00,116 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:05:00,116 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:05:00,117 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:05:00,117 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:05:00,118 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:05:00,119 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:05:00,120 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:05:00,157 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:05:00,157 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:05:00,157 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:05:00,157 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:05:00,158 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:05:00,158 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:05:00,158 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:05:00,158 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:05:00,163 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:05:00,164 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:05:00,164 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:05:00,164 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:05:00,402 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:01,178 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:05:01,178 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:05:01,178 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:05:01,178 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:05:01,179 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:05:01,180 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:05:01,181 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:05:01,182 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:05:01,183 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:05:01,403 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:02,403 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:03,403 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:04,403 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:05,002 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:05:05,404 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:06,404 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:07,404 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:08,404 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:08,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:05:08,989 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:05:09,405 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:10,224 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:05:10,404 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:11,405 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:12,405 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:13,406 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:14,405 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:15,406 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:15,896 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:05:15,896 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:05:15,896 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:05:15,896 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:05:15,897 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:05:15,897 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:05:15,897 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:05:15,897 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:05:15,904 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:05:15,904 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:05:15,904 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:05:15,905 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:05:15,905 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:05:15,937 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:05:15,937 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:05:15,937 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:05:15,937 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:05:15,937 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:05:15,938 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:05:15,938 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:05:15,938 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:05:15,938 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:05:15,938 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:05:15,938 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:05:15,939 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:05:15,940 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:05:16,405 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:17,405 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:18,121 INFO Thread-12 :228 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/output.log 2024-07-24 14:05:18,405 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:19,406 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:20,406 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:21,003 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:05:21,406 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:22,406 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:23,406 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:23,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:05:23,988 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:05:24,406 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:25,406 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:26,230 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:05:26,407 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:27,407 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:28,407 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:29,407 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:30,407 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:31,408 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:32,003 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:05:32,408 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:33,408 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:34,408 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:35,408 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:36,408 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:37,409 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:38,004 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:05:38,409 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:38,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:05:38,989 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:05:39,409 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:40,409 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:41,409 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:42,409 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:43,246 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:05:43,409 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:44,410 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:45,410 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:46,410 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:47,321 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:05:47,321 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:05:47,321 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:05:47,321 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:05:47,322 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:05:47,323 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:05:47,323 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:05:47,324 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:05:47,324 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:05:47,362 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:05:47,362 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:05:47,362 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:05:47,362 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:05:47,362 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:05:47,362 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:05:47,363 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:05:47,363 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:05:47,368 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:05:47,368 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:05:47,369 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:05:47,369 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:05:47,410 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:48,370 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:05:48,410 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:49,411 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:50,411 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:51,411 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:52,411 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:52,683 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:05:52,683 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:05:52,683 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:05:52,683 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:05:52,683 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:05:52,684 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:05:52,685 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:05:52,686 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:05:52,687 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:05:52,724 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:05:52,724 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:05:52,724 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:05:52,724 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:05:52,724 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:05:52,724 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:05:52,725 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:05:52,725 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:05:52,730 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:05:52,731 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:05:52,731 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:05:52,731 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:05:53,233 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:05:53,233 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:05:53,233 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:05:53,233 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:05:53,234 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:05:53,234 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:05:53,235 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:05:53,236 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:05:53,237 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:05:53,246 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:05:53,246 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:05:53,246 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:05:53,246 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:05:53,247 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:05:53,247 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:05:53,247 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:05:53,247 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:05:53,254 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:05:53,254 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:05:53,255 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:05:53,258 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:05:53,411 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:53,988 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:05:53,989 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:05:54,258 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:05:54,411 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:55,411 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:56,412 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:57,412 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:58,412 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:58,564 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:05:58,564 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:05:58,564 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:05:58,564 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:05:58,565 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:05:58,567 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:05:58,567 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:05:58,568 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:05:58,570 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:05:58,605 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:05:58,605 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:05:58,605 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:05:58,605 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:05:58,605 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:05:58,606 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:05:58,606 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:05:58,606 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:05:58,611 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:05:58,611 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:05:58,612 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:05:58,612 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:05:59,412 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:05:59,703 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:05:59,703 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:05:59,704 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:05:59,704 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:05:59,704 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:05:59,705 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:05:59,706 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:05:59,706 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:05:59,707 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:06:00,004 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:06:00,413 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:01,413 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:02,413 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:03,413 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:04,413 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:05,005 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:06:05,414 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:06,413 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:07,414 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:08,414 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:08,989 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:06:08,990 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:06:09,414 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:10,253 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:06:10,415 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:11,414 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:12,415 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:13,415 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:14,365 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:06:14,365 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:06:14,365 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:06:14,366 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:06:14,366 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:06:14,366 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:06:14,367 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:06:14,368 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:06:14,378 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:06:14,379 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:06:14,379 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:06:14,379 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:06:14,406 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:06:14,406 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:06:14,406 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:06:14,406 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:06:14,406 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:06:14,407 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:06:14,407 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:06:14,407 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:06:14,408 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:06:14,408 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:06:14,409 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:06:14,409 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:06:14,409 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:06:14,410 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:06:14,410 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:06:14,411 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:06:14,413 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:06:14,419 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:06:14,419 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:06:14,419 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:06:14,420 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:14,420 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:06:15,415 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:15,420 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:06:16,143 INFO Thread-12 :228 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/output.log 2024-07-24 14:06:16,415 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:17,415 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:18,415 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:19,415 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:20,415 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:21,004 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:06:21,416 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:22,416 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:23,416 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:23,989 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:06:23,989 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:06:24,416 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:25,416 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:26,263 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:06:26,416 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:27,128 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:06:27,128 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:06:27,129 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:06:27,129 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:06:27,129 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:06:27,130 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:06:27,131 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:06:27,133 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:06:27,134 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:06:27,417 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:28,417 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:29,419 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:30,417 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:31,418 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:32,006 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:06:32,250 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:06:32,250 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:06:32,250 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:06:32,251 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:06:32,251 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:06:32,251 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:06:32,252 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:06:32,252 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:06:32,258 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:06:32,259 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:06:32,259 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:06:32,259 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:06:32,291 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:06:32,291 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:06:32,291 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:06:32,291 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:06:32,291 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:06:32,292 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:06:32,292 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:06:32,292 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:06:32,292 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:06:32,292 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:06:32,292 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:06:32,293 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:06:32,294 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:06:32,417 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:33,417 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:34,150 INFO Thread-12 :228 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/output.log 2024-07-24 14:06:34,417 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:35,418 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:36,418 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:37,418 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:38,005 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:06:38,418 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:38,989 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:06:38,989 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:06:39,418 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:40,419 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:41,419 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:42,419 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:43,214 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:06:43,419 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:44,419 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:45,419 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:46,420 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:47,420 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:48,420 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:49,005 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:06:49,420 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:50,420 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:51,420 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:52,420 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:53,421 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:53,989 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:06:53,990 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:06:54,263 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:06:54,421 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:55,421 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:56,421 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:57,421 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:58,422 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:06:59,422 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:00,005 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:07:00,422 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:01,422 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:02,422 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:03,422 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:04,422 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:05,423 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:06,006 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:07:06,423 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:07,423 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:08,423 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:08,989 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:07:08,990 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:07:09,424 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:10,424 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:11,297 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:07:11,424 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:12,424 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:13,424 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:14,424 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:15,424 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:16,425 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:17,006 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:07:17,425 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:18,425 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:19,425 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:20,425 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:21,425 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:22,006 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:07:22,425 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:23,426 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:23,989 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:07:23,990 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:07:24,426 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:25,426 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:26,426 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:27,277 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:07:27,426 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:28,427 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:29,427 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:30,427 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:31,427 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:32,427 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:33,007 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:07:33,427 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:34,428 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:35,428 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:36,428 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:37,428 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:38,007 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:07:38,428 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:38,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:07:38,990 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:07:39,428 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:40,429 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:41,429 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:42,429 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:43,222 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:07:43,429 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:44,429 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:45,429 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:46,429 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:47,429 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:48,430 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:49,007 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:07:49,430 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:50,430 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:51,430 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:52,430 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:53,430 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:53,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:07:53,990 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:07:54,189 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:07:54,431 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:55,431 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:56,431 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:57,431 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:58,431 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:07:59,432 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:00,007 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:08:00,432 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:01,432 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:02,432 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:03,432 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:04,432 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:05,433 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:06,008 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:08:06,433 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:07,433 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:08,433 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:08,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:08:08,990 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:08:09,433 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:10,434 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:11,226 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:08:11,434 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:12,434 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:13,434 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:14,434 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:15,435 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:16,435 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:17,008 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:08:17,435 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:18,435 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:19,435 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:20,435 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:21,435 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:22,008 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:08:22,436 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:23,436 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:23,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:08:23,991 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:08:24,436 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:25,436 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:26,436 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:27,241 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:08:27,436 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:28,437 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:29,437 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:30,437 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:31,437 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:32,437 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:33,009 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:08:33,437 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:34,437 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:35,438 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:36,438 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:37,438 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:38,438 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:38,990 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:08:38,991 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:08:39,241 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:08:39,438 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:40,439 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:41,439 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:42,439 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:43,439 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:44,440 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:45,009 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:08:45,440 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:46,440 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:47,440 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:48,440 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:49,440 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:50,440 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:51,009 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:08:51,441 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:52,441 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:53,441 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:53,991 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:08:53,991 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:08:54,441 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:55,442 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:56,214 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:08:56,441 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:57,442 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:58,442 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:08:59,442 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:00,442 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:01,442 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:02,010 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:09:02,443 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:03,443 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:04,443 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:05,443 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:06,443 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:07,010 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:09:07,444 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:08,444 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:08,991 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:09:08,991 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:09:09,444 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:10,444 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:11,444 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:11,605 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:09:11,605 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:09:11,605 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:09:11,605 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:09:11,606 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:09:11,607 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:09:11,607 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:09:11,608 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:09:11,609 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:09:11,646 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:09:11,646 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:09:11,646 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:09:11,646 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:09:11,646 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:09:11,646 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:09:11,647 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:09:11,647 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:09:11,652 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:09:11,652 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:09:11,653 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:09:11,653 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:09:11,823 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:09:11,824 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:09:11,824 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:09:11,824 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:09:11,824 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:09:11,825 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:09:11,825 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:09:11,826 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:09:11,826 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:09:11,826 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:09:11,826 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:09:11,826 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:09:11,826 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:09:11,828 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:09:11,829 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:09:12,444 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:12,537 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:09:12,537 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:09:12,537 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:09:12,537 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:09:12,538 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:09:12,539 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:09:12,541 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:09:12,541 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:09:12,542 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:09:12,578 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:09:12,578 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:09:12,578 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:09:12,578 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:09:12,578 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:09:12,579 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:09:12,579 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:09:12,579 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:09:12,584 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:09:12,584 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:09:12,585 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:09:12,585 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:09:12,585 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:09:13,444 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:14,444 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:15,445 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:16,445 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:17,445 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:18,010 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:09:18,445 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:18,598 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:09:18,598 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:09:18,598 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:09:18,599 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:09:18,599 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:09:18,600 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:09:18,600 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:09:18,601 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:09:18,602 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:09:18,639 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:09:18,639 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:09:18,639 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:09:18,639 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:09:18,639 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:09:18,639 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:09:18,640 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:09:18,640 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:09:18,645 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:09:18,645 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:09:18,645 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:09:18,646 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:09:19,445 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:20,445 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:21,020 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:09:21,020 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:09:21,020 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:09:21,020 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:09:21,021 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:09:21,022 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:09:21,023 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:09:21,024 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:09:21,025 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:09:21,446 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:22,446 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:23,010 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:09:23,446 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:23,991 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:09:23,992 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:09:24,447 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:25,447 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:25,827 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:09:25,827 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:09:25,827 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:09:25,827 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:09:25,828 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:09:25,828 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:09:25,829 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:09:25,829 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:09:25,837 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:09:25,837 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:09:25,837 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:09:25,838 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:09:25,868 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:09:25,868 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:09:25,868 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:09:25,868 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:09:25,868 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:09:25,868 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:09:25,868 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:09:25,869 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:09:25,869 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:09:25,869 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:09:25,869 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:09:25,870 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:09:25,871 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:09:26,447 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:27,447 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:28,213 INFO Thread-12 :228 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/output.log 2024-07-24 14:09:28,447 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:28,872 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:09:29,447 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:30,447 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:31,447 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:32,447 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:33,448 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:34,010 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:09:34,448 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:35,448 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:36,448 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:37,448 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:38,449 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:38,991 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:09:38,992 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:09:39,238 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:09:39,449 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:40,449 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:41,449 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:42,449 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:43,449 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:44,450 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:45,011 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:09:45,450 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:45,528 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:09:45,528 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:09:45,528 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:09:45,529 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:09:45,530 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:09:45,530 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:09:45,532 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:09:45,532 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:09:45,533 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:09:46,450 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:47,450 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:48,450 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:48,563 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:09:48,563 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:09:48,563 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:09:48,564 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:09:48,564 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:09:48,564 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:09:48,564 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:09:48,565 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:09:48,571 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:09:48,571 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:09:48,571 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:09:48,572 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:09:49,450 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:50,450 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:51,011 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:09:51,451 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:52,451 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:53,451 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:53,991 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:09:53,992 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:09:54,451 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:55,451 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:56,259 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:09:56,451 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:57,451 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:58,452 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:09:59,452 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:00,452 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:01,452 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:02,012 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:10:02,452 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:03,452 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:03,512 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:10:03,512 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:10:03,512 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:10:03,512 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:10:03,513 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:10:03,513 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:10:03,514 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:10:03,515 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:10:03,520 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:10:03,553 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:10:03,553 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:10:03,553 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:10:03,553 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:10:03,553 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:10:03,554 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:10:03,554 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:10:03,554 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:10:03,560 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:10:03,560 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:10:03,560 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:10:03,561 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:10:04,262 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:10:04,262 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:10:04,262 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:10:04,262 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:10:04,263 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:10:04,264 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:10:04,266 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:10:04,266 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:10:04,266 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:10:04,303 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:10:04,303 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:10:04,303 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:10:04,303 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:10:04,303 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:10:04,303 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:10:04,304 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:10:04,304 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:10:04,309 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:10:04,309 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:10:04,309 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:10:04,310 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:10:04,452 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:04,754 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:10:04,754 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:10:04,754 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:10:04,754 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:10:04,755 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:10:04,756 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:10:04,756 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:10:04,757 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:10:04,757 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:10:04,809 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:10:04,809 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:10:04,809 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:10:04,810 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:10:04,810 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:10:04,810 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:10:04,811 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:10:04,811 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:10:04,817 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:10:04,817 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:10:04,817 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:10:04,818 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:10:05,453 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:06,453 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:07,453 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:07,819 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:10:08,453 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:08,992 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:10:08,992 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:10:09,453 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:10,034 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:10:10,034 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:10:10,034 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:10:10,035 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:10:10,035 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:10:10,036 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:10:10,037 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:10:10,037 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:10:10,038 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:10:10,075 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:10:10,075 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:10:10,075 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:10:10,075 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:10:10,075 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:10:10,075 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:10:10,076 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:10:10,076 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:10:10,082 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:10:10,082 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:10:10,082 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:10:10,082 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:10:10,454 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:11,206 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:10:11,206 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:10:11,206 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:10:11,206 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:10:11,207 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:10:11,207 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:10:11,212 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:10:11,213 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:10:11,214 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:10:11,454 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:12,454 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:13,083 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:10:13,455 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:14,455 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:15,455 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:16,293 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:10:16,294 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:10:16,294 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:10:16,294 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:10:16,295 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:10:16,295 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:10:16,295 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:10:16,295 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:10:16,302 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:10:16,302 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:10:16,302 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:10:16,303 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:10:16,334 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:10:16,334 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:10:16,334 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:10:16,334 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:10:16,334 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:10:16,335 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:10:16,335 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:10:16,335 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:10:16,335 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:10:16,335 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:10:16,335 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:10:16,336 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:10:16,336 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:10:16,455 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:17,455 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:18,230 INFO Thread-12 :228 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/output.log 2024-07-24 14:10:18,337 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:10:18,455 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:19,455 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:20,455 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:21,456 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:22,456 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:23,456 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:23,992 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:10:23,992 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:10:24,296 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:10:24,456 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:25,456 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:26,456 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:27,456 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:28,457 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:29,457 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:30,013 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:10:30,457 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:31,457 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:32,457 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:33,458 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:34,458 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:35,458 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:36,013 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:10:36,458 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:37,458 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:38,458 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:38,992 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:10:38,992 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:10:39,459 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:40,459 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:41,298 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:10:41,459 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:42,459 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:43,459 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:44,459 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:45,460 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:46,460 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:47,013 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:10:47,460 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:48,460 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:49,460 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:50,460 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:51,461 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:52,014 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:10:52,461 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:53,461 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:53,992 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:10:53,993 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:10:54,461 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:55,461 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:56,461 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:57,305 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:10:57,461 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:58,462 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:10:59,462 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:00,462 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:01,462 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:02,462 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:03,014 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:11:03,462 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:04,462 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:05,463 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:06,463 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:07,463 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:08,463 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:08,992 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:11:08,993 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:11:09,237 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:11:09,463 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:10,464 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:11,464 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:12,464 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:13,464 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:14,464 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:15,014 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:11:15,464 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:16,465 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:17,465 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:18,465 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:19,465 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:20,465 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:21,014 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:11:21,465 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:22,466 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:23,466 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:23,992 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:11:23,993 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:11:24,466 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:25,466 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:26,218 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:11:26,466 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:27,466 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:28,467 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:29,467 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:30,467 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:31,467 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:32,015 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:11:32,467 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:33,467 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:34,468 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:35,468 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:36,468 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:37,468 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:38,015 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:11:38,468 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:38,993 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:11:38,993 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:11:39,468 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:40,469 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:41,469 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:42,469 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:43,307 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:11:43,469 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:44,470 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:45,470 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:46,470 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:47,470 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:48,470 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:49,015 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:11:49,470 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:50,471 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:51,471 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:52,471 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:53,471 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:53,993 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:11:53,993 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:11:54,283 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:11:54,471 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:55,471 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:56,472 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:57,472 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:58,472 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:11:59,472 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:00,015 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:12:00,472 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:01,473 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:02,473 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:03,473 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:04,473 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:05,015 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:12:05,473 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:06,473 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:07,474 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:08,474 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:08,993 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:12:08,993 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:12:09,474 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:10,334 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:12:10,474 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:11,474 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:12,474 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:13,475 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:14,475 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:15,475 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:16,016 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:12:16,475 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:17,475 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:18,476 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:19,476 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:20,476 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:21,016 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:12:21,476 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:22,476 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:23,477 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:23,993 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:12:23,994 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:12:24,477 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:25,477 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:26,293 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:12:26,477 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:27,477 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:28,477 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:29,477 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:30,477 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:31,478 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:32,017 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:12:32,478 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:33,478 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:34,478 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:35,478 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:36,478 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:37,478 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:38,017 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:12:38,479 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:38,994 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:12:38,994 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:12:39,479 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:40,479 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:41,479 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:42,479 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:43,280 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:12:43,480 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:44,480 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:45,480 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:46,480 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:47,480 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:48,480 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:49,017 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:12:49,480 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:50,481 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:51,481 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:52,481 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:53,481 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:53,994 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:12:53,994 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:12:54,285 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:12:54,481 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:55,482 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:56,482 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:57,482 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:58,482 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:12:59,482 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:00,017 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:13:00,482 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:01,483 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:02,483 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:03,483 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:04,483 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:05,018 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:13:05,483 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:06,483 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:07,484 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:08,484 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:08,994 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:13:08,994 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:13:09,484 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:10,271 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:13:10,484 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:11,484 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:12,484 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:13,485 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:14,485 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:15,485 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:16,018 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:13:16,485 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:17,485 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:18,485 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:19,485 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:20,486 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:21,018 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:13:21,486 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:22,486 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:23,486 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:23,994 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:13:23,994 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:13:24,486 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:25,486 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:26,297 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:13:26,487 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:27,487 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:28,487 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:29,487 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:30,487 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:31,487 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:32,019 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:13:32,487 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:33,488 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:34,488 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:35,488 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:36,488 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:37,019 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:13:37,488 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:38,488 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:38,994 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:13:38,995 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:13:39,489 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:40,489 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:41,489 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:42,267 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:13:42,489 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:43,489 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:44,489 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:45,489 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:46,490 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:47,490 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:48,019 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:13:48,490 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:49,490 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:50,490 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:51,490 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:52,491 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:53,019 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:13:53,491 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:53,994 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:13:53,995 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:13:54,491 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:55,491 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:56,491 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:57,492 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:58,297 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:13:58,492 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:13:59,492 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:00,492 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:01,492 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:02,492 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:03,493 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:04,019 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:14:04,493 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:05,493 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:06,493 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:07,493 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:08,493 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:08,994 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:14:08,995 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:14:09,245 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:14:09,493 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:10,494 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:11,494 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:12,494 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:13,494 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:14,494 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:15,020 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:14:15,494 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:16,495 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:17,495 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:18,495 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:19,495 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:20,020 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:14:20,495 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:21,495 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:22,495 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:23,496 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:23,995 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:14:23,995 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:14:24,496 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:25,211 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:14:25,496 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:26,496 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:27,496 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:28,496 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:29,497 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:30,497 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:31,020 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:14:31,497 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:32,497 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:33,497 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:34,498 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:35,498 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:36,021 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:14:36,498 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:37,498 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:38,498 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:38,995 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:14:38,995 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:14:39,498 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:40,499 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:41,266 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:14:41,499 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:42,499 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:43,499 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:44,499 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:45,499 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:46,500 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:47,021 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:14:47,500 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:48,500 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:49,500 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:50,501 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:51,501 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:52,501 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:53,021 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:14:53,501 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:53,995 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:14:53,995 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:14:54,501 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:55,501 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:56,501 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:57,502 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:58,293 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:14:58,502 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:14:59,502 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:00,502 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:01,502 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:02,502 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:03,503 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:04,021 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:15:04,503 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:05,503 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:06,503 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:07,503 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:08,504 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:08,995 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:15:08,996 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:15:09,217 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:15:09,504 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:10,504 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:11,504 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:12,504 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:13,504 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:14,504 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:15,022 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:15:15,505 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:16,505 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:17,505 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:18,505 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:19,505 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:20,022 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:15:20,505 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:21,505 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:22,506 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:23,506 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:23,995 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:15:23,996 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:15:24,506 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:25,278 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:15:25,506 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:26,506 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:27,506 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:28,507 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:29,507 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:30,507 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:31,023 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:15:31,507 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:32,507 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:33,507 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:34,508 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:35,508 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:36,023 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:15:36,508 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:37,508 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:38,508 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:38,995 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:15:38,996 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:15:39,508 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:40,509 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:41,253 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:15:41,509 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:42,509 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:43,509 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:44,509 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:45,510 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:46,510 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:47,023 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:15:47,510 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:48,510 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:49,510 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:50,510 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:51,511 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:52,511 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:53,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:15:53,511 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:53,996 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:15:53,996 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:15:54,511 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:55,511 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:56,511 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:57,511 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:58,284 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:15:58,512 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:15:59,512 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:00,512 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:01,512 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:02,512 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:03,512 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:04,023 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:16:04,512 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:05,513 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:06,513 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:07,513 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:08,513 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:08,996 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:16:08,996 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:16:09,243 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:16:09,513 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:10,513 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:11,514 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:12,514 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:13,514 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:14,514 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:15,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:16:15,514 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:16,514 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:17,514 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:18,515 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:19,515 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:20,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:16:20,556 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:21,540 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:22,540 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:23,541 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:23,996 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:16:23,996 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:16:24,541 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:25,320 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:16:25,541 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:26,541 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:27,541 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:28,542 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:29,542 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:30,542 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:31,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:16:31,542 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:32,542 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:33,542 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:34,543 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:35,543 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:36,024 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:16:36,543 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:37,543 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:38,543 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:38,996 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:16:38,997 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:16:39,543 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:40,543 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:41,213 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:16:41,544 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:42,544 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:43,544 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:44,544 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:45,544 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:46,544 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:47,025 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:16:47,544 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:48,545 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:49,545 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:49,825 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:16:49,825 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:16:49,825 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:16:49,825 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:16:49,826 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:16:49,827 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:16:49,827 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:16:49,828 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:16:49,829 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:16:49,866 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:16:49,866 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:16:49,866 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:16:49,866 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:16:49,866 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:16:49,867 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:16:49,867 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:16:49,867 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:16:49,873 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:16:49,873 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:16:49,873 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:16:49,874 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:16:50,545 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:51,153 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:16:51,153 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:16:51,153 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:16:51,154 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:16:51,154 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:16:51,155 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:16:51,156 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:16:51,157 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:16:51,158 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:16:51,545 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:52,076 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:16:52,077 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:16:52,077 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:16:52,077 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:16:52,077 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:16:52,077 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:16:52,078 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:16:52,078 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:16:52,085 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:16:52,085 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:16:52,085 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:16:52,086 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:16:52,086 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:16:52,545 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:53,546 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:53,996 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:16:53,997 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:16:54,546 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:55,546 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:56,546 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:57,221 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:16:57,546 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:58,546 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:16:59,546 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:00,547 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:01,547 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:02,547 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:03,026 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:17:03,547 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:04,547 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:05,548 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:06,548 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:07,548 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:08,548 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:08,996 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:17:08,997 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:17:09,244 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:17:09,548 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:10,548 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:11,549 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:12,549 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:13,549 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:14,549 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:15,025 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:17:15,549 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:16,549 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:17,550 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:18,550 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:19,550 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:20,026 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:17:20,550 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:21,550 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:22,550 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:23,550 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:23,997 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:17:23,997 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:17:24,551 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:25,229 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:17:25,551 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:26,551 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:27,551 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:28,551 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:29,552 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:30,552 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:31,026 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:17:31,552 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:32,552 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:33,552 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:34,552 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:35,553 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:36,026 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:17:36,553 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:37,553 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:38,553 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:38,997 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:17:38,997 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:17:39,553 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:40,553 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:41,287 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:17:41,553 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:42,554 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:43,554 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:44,554 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:45,554 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:46,554 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:47,027 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:17:47,554 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:48,555 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:49,555 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:50,555 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:51,555 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:52,006 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:17:52,006 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:17:52,006 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:17:52,006 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:17:52,007 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:17:52,009 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:17:52,010 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:17:52,011 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:17:52,012 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:17:52,555 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:53,027 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:17:53,557 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:53,997 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:17:53,997 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:17:54,555 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:54,835 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:17:54,836 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:17:54,836 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:17:54,836 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:17:54,836 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:17:54,836 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:17:54,837 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:17:54,837 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:17:54,843 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:17:54,843 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:17:54,843 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:17:54,844 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:17:55,556 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:56,556 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:57,556 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:58,556 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:17:58,845 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:17:59,556 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:00,351 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:18:00,351 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:18:00,351 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:18:00,351 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:18:00,352 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:18:00,353 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:18:00,353 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:18:00,354 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:18:00,355 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:18:00,402 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:18:00,402 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:18:00,402 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:18:00,402 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:18:00,402 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:18:00,403 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:18:00,404 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:18:00,404 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:18:00,411 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:18:00,411 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:18:00,411 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:18:00,412 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:18:00,556 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:01,556 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:02,557 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:03,557 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:04,027 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:18:04,557 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:05,557 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:05,986 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:18:05,986 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:18:05,986 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:18:05,987 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:18:05,987 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:18:05,988 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:18:05,990 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:18:05,991 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:18:05,992 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:18:06,027 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:18:06,027 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:18:06,027 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:18:06,027 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:18:06,027 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:18:06,028 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:18:06,028 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:18:06,028 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:18:06,034 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:18:06,034 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:18:06,034 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:18:06,035 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:18:06,557 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:07,557 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:08,558 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:08,997 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:18:08,997 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:18:09,209 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:18:09,558 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:10,558 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:11,558 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:12,558 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:13,558 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:14,559 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:15,028 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:18:15,559 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:16,559 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:17,559 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:18,559 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:19,560 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:20,560 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:21,028 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:18:21,560 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:22,560 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:23,560 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:23,997 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:18:23,998 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:18:24,560 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:25,560 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:26,306 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:18:26,560 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:27,161 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:18:27,161 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:18:27,161 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:18:27,162 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:18:27,162 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:18:27,163 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:18:27,163 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:18:27,164 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:18:27,165 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:18:27,207 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:18:27,207 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:18:27,207 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:18:27,207 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:18:27,208 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:18:27,208 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:18:27,208 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:18:27,209 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:18:27,215 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:18:27,215 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:18:27,215 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:18:27,216 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:18:27,560 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:28,561 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:29,561 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:30,561 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:31,561 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:32,028 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:18:32,561 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:33,561 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:34,561 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:34,798 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:18:34,798 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:18:34,798 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:18:34,799 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:18:34,799 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:18:34,800 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:18:34,801 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:18:34,801 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:18:34,804 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:18:34,839 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:18:34,839 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:18:34,839 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:18:34,839 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:18:34,839 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:18:34,840 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:18:34,840 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:18:34,840 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:18:34,845 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:18:34,846 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:18:34,846 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:18:34,846 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:18:35,562 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:36,562 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:37,562 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:37,847 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:18:38,562 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:38,997 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:18:38,998 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:18:39,562 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:40,563 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:41,563 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:42,563 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:43,214 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:18:43,214 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:18:43,214 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:18:43,214 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:18:43,215 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:18:43,215 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:18:43,216 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:18:43,217 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:18:43,218 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:18:43,238 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:18:43,563 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:43,833 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: pause 2024-07-24 14:18:43,833 INFO HandlerThread:228 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-24 14:18:43,833 INFO HandlerThread:228 [system_monitor.py:finish():203] Stopping system monitor 2024-07-24 14:18:43,834 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:18:43,834 DEBUG SystemMonitor:228 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-24 14:18:43,834 DEBUG SystemMonitor:228 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-24 14:18:43,835 INFO HandlerThread:228 [interfaces.py:finish():200] Joined cpu monitor 2024-07-24 14:18:43,835 INFO HandlerThread:228 [interfaces.py:finish():200] Joined disk monitor 2024-07-24 14:18:43,842 INFO HandlerThread:228 [interfaces.py:finish():200] Joined gpu monitor 2024-07-24 14:18:43,843 INFO HandlerThread:228 [interfaces.py:finish():200] Joined memory monitor 2024-07-24 14:18:43,843 INFO HandlerThread:228 [interfaces.py:finish():200] Joined network monitor 2024-07-24 14:18:43,843 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:18:44,563 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:45,564 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:46,422 INFO Thread-12 :228 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/output.log 2024-07-24 14:18:46,564 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:47,564 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:47,607 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: resume 2024-07-24 14:18:47,607 INFO HandlerThread:228 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-24 14:18:47,607 INFO HandlerThread:228 [system_monitor.py:start():194] Starting system monitor 2024-07-24 14:18:47,607 INFO SystemMonitor:228 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-24 14:18:47,608 INFO SystemMonitor:228 [interfaces.py:start():188] Started cpu monitoring 2024-07-24 14:18:47,609 INFO SystemMonitor:228 [interfaces.py:start():188] Started disk monitoring 2024-07-24 14:18:47,609 INFO SystemMonitor:228 [interfaces.py:start():188] Started gpu monitoring 2024-07-24 14:18:47,610 INFO SystemMonitor:228 [interfaces.py:start():188] Started memory monitoring 2024-07-24 14:18:47,611 INFO SystemMonitor:228 [interfaces.py:start():188] Started network monitoring 2024-07-24 14:18:48,363 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:18:48,371 DEBUG SenderThread:228 [sender.py:send():379] send: config 2024-07-24 14:18:48,372 DEBUG SenderThread:228 [sender.py:send():379] send: telemetry 2024-07-24 14:18:48,373 DEBUG SenderThread:228 [sender.py:send():379] send: metric 2024-07-24 14:18:48,373 DEBUG SenderThread:228 [sender.py:send():379] send: telemetry 2024-07-24 14:18:48,373 DEBUG SenderThread:228 [sender.py:send():379] send: metric 2024-07-24 14:18:48,373 WARNING SenderThread:228 [sender.py:send_metric():1405] Seen metric with glob (shouldn't happen) 2024-07-24 14:18:48,374 DEBUG SenderThread:228 [sender.py:send():379] send: telemetry 2024-07-24 14:18:48,564 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:48,685 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: log_artifact 2024-07-24 14:18:48,685 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: log_artifact 2024-07-24 14:18:48,915 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: log_artifact 2024-07-24 14:18:49,564 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:50,424 INFO Thread-12 :228 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/output.log 2024-07-24 14:18:50,564 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:51,347 INFO wandb-upload_0:228 [upload_job.py:push():88] Uploaded file /tmp/tmp14m1i44b/model_architecture.txt 2024-07-24 14:18:51,567 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:52,564 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:53,069 INFO SenderThread:228 [sender.py:send_request_log_artifact():1518] logged artifact model-jv4m57pj - {'id': 'QXJ0aWZhY3Q6MTAzMTA5MDY1Mg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzAyNTEyMw==', 'latestArtifact': None}} 2024-07-24 14:18:53,069 DEBUG SenderThread:228 [sender.py:send():379] send: config 2024-07-24 14:18:53,070 DEBUG SenderThread:228 [sender.py:send():379] send: metric 2024-07-24 14:18:53,070 DEBUG SenderThread:228 [sender.py:send():379] send: metric 2024-07-24 14:18:53,070 WARNING SenderThread:228 [sender.py:send_metric():1405] Seen metric with glob (shouldn't happen) 2024-07-24 14:18:53,070 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: log_artifact 2024-07-24 14:18:53,296 INFO SenderThread:228 [sender.py:send_request_log_artifact():1518] logged artifact model-jv4m57pj - {'id': 'QXJ0aWZhY3Q6MTAzMTA5MDY1Mg==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzAyNTEyMw==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTAzMTA5MDY1Mg==', 'versionIndex': 0}}} 2024-07-24 14:18:53,565 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:53,998 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:18:53,998 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:18:54,203 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:18:54,565 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:55,565 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:56,565 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:57,565 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:58,565 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:18:59,565 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:00,029 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:19:00,565 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:01,566 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:02,566 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:03,566 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:04,566 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:05,566 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:06,029 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:19:06,566 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:07,567 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:08,567 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:08,998 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:19:08,998 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:19:09,567 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:10,567 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:11,298 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:19:11,567 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:12,567 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:13,567 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:14,568 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:15,568 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:16,568 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:17,035 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:19:17,434 INFO Thread-12 :228 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/config.yaml 2024-07-24 14:19:17,568 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:18,568 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:19,568 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:20,569 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:21,569 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:22,569 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:23,030 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:19:23,569 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:23,998 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:19:23,998 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:19:24,569 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:25,569 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:26,569 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:27,569 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:28,244 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:19:28,570 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:29,616 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:30,615 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:31,615 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:32,619 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:33,617 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:19:34,363 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:38,618 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:19:39,167 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:39,168 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:19:39,168 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:19:41,678 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:44,487 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:19:47,612 DEBUG SystemMonitor:228 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-24 14:19:47,613 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:19:47,941 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:48,657 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:49,466 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:49,920 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:19:50,414 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:51,414 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:52,414 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:53,415 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:54,168 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:19:54,168 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:19:54,415 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:55,415 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:55,428 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:19:56,415 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:57,415 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:58,415 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:19:59,415 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:00,416 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:01,212 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:20:01,416 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:02,416 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:03,416 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:03,824 DEBUG SenderThread:228 [sender.py:send():379] send: telemetry 2024-07-24 14:20:03,864 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: partial_history 2024-07-24 14:20:03,866 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: partial_history 2024-07-24 14:20:03,867 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: summary_record 2024-07-24 14:20:03,869 INFO SenderThread:228 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-24 14:20:03,870 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: summary_record 2024-07-24 14:20:04,416 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:04,453 INFO Thread-12 :228 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/wandb-summary.json 2024-07-24 14:20:05,417 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:06,417 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:07,417 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:08,417 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:09,167 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:20:09,418 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:10,418 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:11,419 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:12,419 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:13,419 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:14,419 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:15,419 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:16,419 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:17,419 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:17,533 INFO SenderThread:228 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-24 14:20:17,534 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: summary_record 2024-07-24 14:20:17,535 INFO SenderThread:228 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-24 14:20:17,535 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: summary_record 2024-07-24 14:20:17,535 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:20:17,575 INFO SenderThread:228 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-24 14:20:17,575 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: summary_record 2024-07-24 14:20:17,576 INFO SenderThread:228 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-24 14:20:17,576 DEBUG SenderThread:228 [sender.py:send():379] send: metric 2024-07-24 14:20:17,576 DEBUG SenderThread:228 [sender.py:send():379] send: history 2024-07-24 14:20:17,577 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: summary_record 2024-07-24 14:20:17,743 INFO SenderThread:228 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-24 14:20:17,744 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: summary_record 2024-07-24 14:20:17,745 INFO SenderThread:228 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-24 14:20:17,745 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: summary_record 2024-07-24 14:20:17,792 INFO SenderThread:228 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-24 14:20:17,792 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: summary_record 2024-07-24 14:20:17,793 INFO SenderThread:228 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-24 14:20:17,793 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: summary_record 2024-07-24 14:20:17,797 INFO SenderThread:228 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-24 14:20:17,798 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: summary_record 2024-07-24 14:20:17,826 INFO SenderThread:228 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-24 14:20:17,827 DEBUG SenderThread:228 [sender.py:send():379] send: history 2024-07-24 14:20:17,827 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: summary_record 2024-07-24 14:20:17,827 INFO SenderThread:228 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-24 14:20:17,828 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:20:18,075 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:20:18,419 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:18,459 INFO Thread-12 :228 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/wandb-summary.json 2024-07-24 14:20:19,420 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:20,420 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:21,420 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:22,420 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:23,082 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:20:23,420 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:24,168 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:20:24,168 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:20:24,420 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:24,461 INFO Thread-12 :228 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240724_135436-jv4m57pj/files/config.yaml 2024-07-24 14:20:25,421 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:26,421 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:27,421 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:28,421 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:29,405 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:20:29,421 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:30,421 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:31,422 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:32,422 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:33,422 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:34,406 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:20:34,422 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:35,422 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:36,422 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:37,423 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:38,423 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:39,168 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:20:39,168 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:20:39,423 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:39,443 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:20:40,423 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:41,425 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:42,425 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:43,425 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:44,425 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:44,444 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:20:45,425 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:46,425 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:47,425 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:47,615 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:20:48,426 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:49,426 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:49,616 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:20:50,426 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:51,426 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:52,426 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:53,426 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:54,168 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:20:54,169 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:20:54,426 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:55,391 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:20:55,427 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:56,427 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:57,427 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:58,427 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:20:59,427 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:00,392 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:21:00,427 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:01,428 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:02,428 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:03,428 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:04,428 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:05,393 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:21:05,429 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:06,429 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:07,429 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:08,429 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:09,168 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:21:09,169 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:21:09,429 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:10,418 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:21:10,429 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:11,429 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:12,429 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:13,430 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:14,430 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:15,419 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:21:15,430 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:16,430 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:17,430 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:17,616 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:21:18,431 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:19,431 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:20,431 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:20,617 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:21:21,431 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:22,431 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:23,431 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:24,168 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:21:24,169 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:21:24,431 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:25,432 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:26,432 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:26,438 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:21:27,432 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:28,432 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:29,432 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:30,432 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:31,434 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:31,439 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:21:32,433 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:33,433 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:34,433 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:35,433 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:36,433 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:36,440 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:21:37,434 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:38,434 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:39,169 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:21:39,169 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:21:39,434 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:40,434 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:41,435 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:41,482 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:21:42,434 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:43,435 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:44,435 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:45,435 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:46,435 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:46,483 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:21:47,435 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:47,617 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:21:48,436 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:49,436 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:50,436 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:51,436 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:51,618 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:21:52,436 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:53,436 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:54,169 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:21:54,169 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:21:54,437 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:55,437 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:56,437 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:57,437 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:57,456 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:21:58,438 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:21:59,438 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:00,438 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:01,438 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:02,438 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:02,457 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:22:03,438 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:04,439 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:05,439 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:06,439 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:07,439 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:07,458 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:22:08,440 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:09,169 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:22:09,170 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:22:09,440 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:10,440 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:11,440 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:12,440 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:13,441 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:13,445 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:22:14,441 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:15,441 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:16,441 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:17,442 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:17,617 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:22:18,442 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:18,618 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:22:19,442 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:20,442 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:21,442 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:22,443 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:23,443 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:23,619 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:22:24,169 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:22:24,170 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:22:24,443 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:25,443 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:26,443 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:27,443 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:28,444 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:29,396 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:22:29,444 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:30,444 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:31,444 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:32,444 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:33,444 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:34,397 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:22:34,445 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:35,445 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:36,445 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:37,445 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:38,445 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:39,170 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: stop_status 2024-07-24 14:22:39,170 DEBUG SenderThread:228 [sender.py:send_request():406] send_request: stop_status 2024-07-24 14:22:39,445 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:39,451 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:22:40,446 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:41,446 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:42,446 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:43,446 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:44,446 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:44,451 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report 2024-07-24 14:22:45,447 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:46,446 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:47,447 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:47,618 DEBUG SenderThread:228 [sender.py:send():379] send: stats 2024-07-24 14:22:48,447 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:49,447 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-24 14:22:49,619 DEBUG HandlerThread:228 [handler.py:handle_request():158] handle_request: status_report