2024-06-30 07:58:20,866 INFO StreamThr :191 [internal.py:wandb_internal():85] W&B internal server running at pid: 191, started at: 2024-06-30 07:58:20.865206 2024-06-30 07:58:20,867 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status 2024-06-30 07:58:21,519 INFO WriterThread:191 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240630_075820-qdco2zur/run-qdco2zur.wandb 2024-06-30 07:58:21,520 DEBUG SenderThread:191 [sender.py:send():378] send: header 2024-06-30 07:58:21,523 DEBUG SenderThread:191 [sender.py:send():378] send: run 2024-06-30 07:58:21,887 INFO SenderThread:191 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240630_075820-qdco2zur/files 2024-06-30 07:58:21,887 INFO SenderThread:191 [sender.py:_start_run_threads():1123] run started: qdco2zur with start time 1719734300.866251 2024-06-30 07:58:21,897 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: check_version 2024-06-30 07:58:21,897 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: check_version 2024-06-30 07:58:21,979 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: run_start 2024-06-30 07:58:21,991 DEBUG HandlerThread:191 [system_info.py:__init__():26] System info init 2024-06-30 07:58:21,991 DEBUG HandlerThread:191 [system_info.py:__init__():41] System info init done 2024-06-30 07:58:21,991 INFO HandlerThread:191 [system_monitor.py:start():194] Starting system monitor 2024-06-30 07:58:21,991 INFO SystemMonitor:191 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-30 07:58:21,992 INFO HandlerThread:191 [system_monitor.py:probe():214] Collecting system info 2024-06-30 07:58:21,992 INFO SystemMonitor:191 [interfaces.py:start():188] Started cpu monitoring 2024-06-30 07:58:21,993 INFO SystemMonitor:191 [interfaces.py:start():188] Started disk monitoring 2024-06-30 07:58:21,994 INFO SystemMonitor:191 [interfaces.py:start():188] Started gpu monitoring 2024-06-30 07:58:21,995 INFO SystemMonitor:191 [interfaces.py:start():188] Started memory monitoring 2024-06-30 07:58:21,995 INFO SystemMonitor:191 [interfaces.py:start():188] Started network monitoring 2024-06-30 07:58:22,008 DEBUG HandlerThread:191 [system_info.py:probe():150] Probing system 2024-06-30 07:58:22,010 DEBUG HandlerThread:191 [gitlib.py:_init_repo():56] git repository is invalid 2024-06-30 07:58:22,010 DEBUG HandlerThread:191 [system_info.py:probe():198] Probing system done 2024-06-30 07:58:22,010 DEBUG HandlerThread:191 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-06-30T07:58:22.008256', 'startedAt': '2024-06-30T07:58:20.858894', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'fe0981c45f21', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.182, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.182, 'min': 0.0, 'max': 0.0}, {'current': 2000.182, 'min': 0.0, 'max': 0.0}, {'current': 2000.182, 'min': 0.0, 'max': 0.0}, {'current': 2000.182, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5689.846576690674}}, 'gpu': 'Tesla P100-PCIE-16GB', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla P100-PCIE-16GB', 'memory_total': 17179869184}], 'memory': {'total': 31.357563018798828}} 2024-06-30 07:58:22,010 INFO HandlerThread:191 [system_monitor.py:probe():224] Finished collecting system info 2024-06-30 07:58:22,010 INFO HandlerThread:191 [system_monitor.py:probe():227] Publishing system info 2024-06-30 07:58:22,010 DEBUG HandlerThread:191 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment 2024-06-30 07:58:22,889 INFO Thread-12 :191 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240630_075820-qdco2zur/files/conda-environment.yaml 2024-06-30 07:58:37,024 ERROR HandlerThread:191 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda subprocess.call( File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call return p.wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait return self._wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait raise TimeoutExpired(self.args, timeout) subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds 2024-06-30 07:58:37,027 DEBUG HandlerThread:191 [system_info.py:_save_conda():222] Saving conda packages done 2024-06-30 07:58:37,027 INFO HandlerThread:191 [system_monitor.py:probe():229] Finished publishing system info 2024-06-30 07:58:37,035 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:58:37,035 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: keepalive 2024-06-30 07:58:37,035 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:58:37,035 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: keepalive 2024-06-30 07:58:37,035 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:58:37,035 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: keepalive 2024-06-30 07:58:37,036 DEBUG SenderThread:191 [sender.py:send():378] send: files 2024-06-30 07:58:37,036 INFO SenderThread:191 [sender.py:_save_file():1389] saving file wandb-metadata.json with policy now 2024-06-30 07:58:37,633 INFO wandb-upload_0:191 [upload_job.py:push():130] Uploaded file /tmp/tmp8wajj86owandb/7holgybb-wandb-metadata.json 2024-06-30 07:58:37,892 INFO Thread-12 :191 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240630_075820-qdco2zur/files/wandb-metadata.json 2024-06-30 07:58:37,998 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: python_packages 2024-06-30 07:58:37,998 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: python_packages 2024-06-30 07:58:38,001 DEBUG SenderThread:191 [sender.py:send():378] send: telemetry 2024-06-30 07:58:38,002 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 07:58:38,002 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 07:58:38,148 DEBUG SenderThread:191 [sender.py:send():378] send: config 2024-06-30 07:58:38,150 DEBUG SenderThread:191 [sender.py:send():378] send: metric 2024-06-30 07:58:38,150 DEBUG SenderThread:191 [sender.py:send():378] send: telemetry 2024-06-30 07:58:38,150 DEBUG SenderThread:191 [sender.py:send():378] send: telemetry 2024-06-30 07:58:38,150 DEBUG SenderThread:191 [sender.py:send():378] send: metric 2024-06-30 07:58:38,150 WARNING SenderThread:191 [sender.py:send_metric():1340] Seen metric with glob (shouldn't happen) 2024-06-30 07:58:38,150 DEBUG SenderThread:191 [sender.py:send():378] send: telemetry 2024-06-30 07:58:38,892 INFO Thread-12 :191 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240630_075820-qdco2zur/files/requirements.txt 2024-06-30 07:58:41,151 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:58:46,152 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:58:51,158 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:58:51,896 INFO Thread-12 :191 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240630_075820-qdco2zur/files/config.yaml 2024-06-30 07:58:53,021 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 07:58:53,022 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 07:58:57,245 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:59:02,246 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:59:07,247 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:59:08,020 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 07:59:08,021 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 07:59:12,255 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:59:17,256 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:59:21,996 DEBUG SystemMonitor:191 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-30 07:59:21,997 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 07:59:22,998 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:59:23,167 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 07:59:23,167 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 07:59:28,329 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:59:33,330 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:59:38,166 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 07:59:38,167 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 07:59:38,332 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:59:43,333 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:59:48,334 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:59:51,998 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 07:59:53,166 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 07:59:53,167 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 07:59:53,364 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 07:59:58,365 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:00:03,366 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:00:08,342 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:00:08,343 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:00:08,476 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:00:13,477 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:00:18,478 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:00:21,999 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:00:23,341 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:00:23,342 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:00:23,560 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:00:28,561 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:00:33,562 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:00:38,341 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:00:38,342 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:00:39,512 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:00:44,513 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:00:49,514 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:00:52,000 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:00:53,498 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:00:53,499 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:00:54,672 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:00:59,673 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:01:04,674 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:01:08,498 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:01:08,499 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:01:09,719 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:01:14,720 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:01:19,721 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:01:22,001 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:01:23,498 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:01:23,499 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:01:24,726 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:01:29,727 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:01:34,728 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:01:38,694 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:01:38,695 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:01:39,865 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:01:44,866 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:01:49,867 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:01:52,002 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:01:53,693 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:01:53,693 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:01:54,929 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:01:59,930 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:02:04,931 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:02:08,693 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:02:08,694 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:02:09,937 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:02:14,938 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:02:19,939 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:02:22,003 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:02:23,836 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:02:23,836 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:02:25,046 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:02:30,047 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:02:35,048 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:02:38,835 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:02:38,836 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:02:40,997 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:02:45,998 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:02:50,999 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:02:52,003 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:02:53,836 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:02:53,836 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:02:56,999 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:02:57,406 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: partial_history 2024-06-30 08:02:57,409 DEBUG SenderThread:191 [sender.py:send():378] send: metric 2024-06-30 08:02:57,409 DEBUG SenderThread:191 [sender.py:send():378] send: metric 2024-06-30 08:02:57,409 DEBUG SenderThread:191 [sender.py:send():378] send: metric 2024-06-30 08:02:57,410 DEBUG SenderThread:191 [sender.py:send():378] send: metric 2024-06-30 08:02:57,410 DEBUG SenderThread:191 [sender.py:send():378] send: metric 2024-06-30 08:02:57,410 DEBUG SenderThread:191 [sender.py:send():378] send: history 2024-06-30 08:02:57,410 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: summary_record 2024-06-30 08:02:57,411 INFO SenderThread:191 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-30 08:02:57,983 INFO Thread-12 :191 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240630_075820-qdco2zur/files/wandb-summary.json 2024-06-30 08:03:02,412 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:03:07,413 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:03:09,023 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:03:09,024 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:03:13,253 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:03:18,254 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:03:22,004 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:03:24,005 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:03:24,023 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:03:24,024 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:03:29,232 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:03:29,995 INFO Thread-12 :191 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240630_075820-qdco2zur/files/config.yaml 2024-06-30 08:03:34,459 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:03:39,023 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:03:39,024 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:03:40,225 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:03:45,226 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:03:50,227 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:03:52,005 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:03:54,189 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:03:54,189 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:03:55,388 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:04:00,389 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:04:05,390 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:04:09,188 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:04:09,189 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:04:11,365 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:04:16,366 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:04:21,367 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:04:22,006 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:04:24,188 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:04:24,189 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:04:27,363 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:04:32,364 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:04:37,365 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:04:39,365 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:04:39,366 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:04:42,527 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:04:47,528 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:04:52,007 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:04:53,008 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:04:54,365 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:04:54,366 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:04:58,493 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:05:03,494 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:05:08,495 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:05:09,365 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:05:09,366 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:05:13,550 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:05:18,551 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:05:22,008 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:05:24,009 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:05:24,577 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:05:24,577 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:05:29,793 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:05:34,794 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:05:39,576 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:05:39,577 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:05:40,756 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:05:45,757 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:05:50,758 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:05:52,009 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:05:54,576 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:05:54,577 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:05:56,702 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:06:01,703 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:06:06,704 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:06:09,743 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:06:09,744 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:06:11,956 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:06:16,957 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:06:21,958 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:06:22,009 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:06:24,748 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:06:24,749 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:06:25,938 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: partial_history 2024-06-30 08:06:25,940 DEBUG SenderThread:191 [sender.py:send():378] send: metric 2024-06-30 08:06:25,940 DEBUG SenderThread:191 [sender.py:send():378] send: metric 2024-06-30 08:06:25,940 DEBUG SenderThread:191 [sender.py:send():378] send: metric 2024-06-30 08:06:25,940 DEBUG SenderThread:191 [sender.py:send():378] send: history 2024-06-30 08:06:25,940 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: summary_record 2024-06-30 08:06:25,941 INFO SenderThread:191 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-30 08:06:26,056 INFO Thread-12 :191 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240630_075820-qdco2zur/files/wandb-summary.json 2024-06-30 08:06:27,941 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:06:32,942 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:06:37,949 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:06:39,061 INFO Thread-12 :191 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240630_075820-qdco2zur/files/config.yaml 2024-06-30 08:06:40,008 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:06:40,009 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:06:43,191 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:06:48,191 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:06:52,010 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:06:54,011 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:06:55,007 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:06:55,007 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:06:59,244 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:07:04,245 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:07:09,246 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:07:10,007 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:07:10,008 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:07:15,221 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:07:18,786 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: partial_history 2024-06-30 08:07:18,787 DEBUG SenderThread:191 [sender.py:send():378] send: history 2024-06-30 08:07:18,787 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: summary_record 2024-06-30 08:07:18,788 INFO SenderThread:191 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-30 08:07:19,076 INFO Thread-12 :191 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240630_075820-qdco2zur/files/wandb-summary.json 2024-06-30 08:07:20,789 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:07:22,011 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:07:25,195 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:07:25,196 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:07:26,433 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:07:31,434 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:07:36,435 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:07:40,195 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:07:40,195 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:07:42,349 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:07:47,350 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:07:52,012 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:07:53,012 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:07:55,195 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:07:55,196 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:07:58,432 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:08:03,433 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:08:08,434 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:08:10,348 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:08:10,348 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:08:13,534 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:08:18,535 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:08:22,012 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:08:24,013 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:08:25,347 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:08:25,348 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:08:29,553 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:08:34,554 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:08:39,555 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:08:40,347 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:08:40,348 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:08:44,584 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:08:49,585 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:08:52,013 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:08:55,014 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:08:55,510 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:08:55,511 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:09:00,689 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:09:05,690 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:09:10,510 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:09:10,511 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:09:10,711 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:09:15,712 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:09:20,713 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:09:22,014 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:09:25,510 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:09:25,510 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:09:26,640 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:09:31,641 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:09:36,642 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:09:40,716 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:09:40,716 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:09:41,942 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:09:46,943 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:09:51,944 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:09:52,014 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:09:55,715 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:09:55,716 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:09:56,956 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:10:01,957 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:10:06,958 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:10:10,715 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:10:10,716 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:10:12,951 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:10:17,952 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:10:22,015 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:10:23,016 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:10:25,913 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:10:25,914 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:10:28,168 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:10:33,169 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:10:38,170 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:10:40,913 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:10:40,914 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:10:44,047 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:10:49,048 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:10:52,016 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:10:55,017 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:10:55,913 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:10:55,914 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:11:00,119 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:11:05,120 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:11:10,120 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:11:10,918 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:11:10,918 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:11:15,153 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:11:20,155 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:11:22,017 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:11:25,917 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:11:25,918 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:11:26,091 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:11:31,092 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:11:36,093 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:11:37,688 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: partial_history 2024-06-30 08:11:37,689 DEBUG SenderThread:191 [sender.py:send():378] send: history 2024-06-30 08:11:37,689 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: summary_record 2024-06-30 08:11:37,689 INFO SenderThread:191 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-30 08:11:37,692 DEBUG SenderThread:191 [sender.py:send():378] send: telemetry 2024-06-30 08:11:37,692 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: summary_record 2024-06-30 08:11:37,694 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: partial_history 2024-06-30 08:11:37,695 INFO SenderThread:191 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-30 08:11:37,695 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: summary_record 2024-06-30 08:11:37,695 INFO SenderThread:191 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-30 08:11:37,696 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: summary_record 2024-06-30 08:11:37,697 INFO SenderThread:191 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-30 08:11:37,697 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: summary_record 2024-06-30 08:11:37,697 INFO SenderThread:191 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-30 08:11:37,698 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: summary_record 2024-06-30 08:11:37,698 INFO SenderThread:191 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-30 08:11:37,699 DEBUG SenderThread:191 [sender.py:send():378] send: history 2024-06-30 08:11:37,699 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: summary_record 2024-06-30 08:11:37,699 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: pause 2024-06-30 08:11:37,699 INFO HandlerThread:191 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-30 08:11:37,699 INFO HandlerThread:191 [system_monitor.py:finish():203] Stopping system monitor 2024-06-30 08:11:37,700 DEBUG SystemMonitor:191 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-30 08:11:37,700 DEBUG SystemMonitor:191 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-30 08:11:37,701 INFO HandlerThread:191 [interfaces.py:finish():200] Joined cpu monitor 2024-06-30 08:11:37,701 INFO SenderThread:191 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-06-30 08:11:37,701 INFO HandlerThread:191 [interfaces.py:finish():200] Joined disk monitor 2024-06-30 08:11:37,707 INFO HandlerThread:191 [interfaces.py:finish():200] Joined gpu monitor 2024-06-30 08:11:37,707 INFO HandlerThread:191 [interfaces.py:finish():200] Joined memory monitor 2024-06-30 08:11:37,707 INFO HandlerThread:191 [interfaces.py:finish():200] Joined network monitor 2024-06-30 08:11:37,707 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:11:38,176 INFO Thread-12 :191 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240630_075820-qdco2zur/files/wandb-summary.json 2024-06-30 08:11:40,917 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:11:40,918 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:11:42,069 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:11:47,070 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:11:52,076 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:11:53,181 INFO Thread-12 :191 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240630_075820-qdco2zur/files/config.yaml 2024-06-30 08:11:55,917 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:11:55,918 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:11:58,058 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:12:03,060 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:12:08,060 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:12:10,917 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:12:10,918 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:12:13,144 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:12:18,145 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:12:23,146 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:12:25,918 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:12:25,919 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:12:29,132 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:12:34,133 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:12:39,133 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:12:40,918 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:12:40,918 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:12:45,114 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:12:50,115 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:12:55,116 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:12:55,918 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:12:55,919 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:13:01,057 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:13:06,059 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:13:10,080 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: resume 2024-06-30 08:13:10,080 INFO HandlerThread:191 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-30 08:13:10,080 INFO HandlerThread:191 [system_monitor.py:start():194] Starting system monitor 2024-06-30 08:13:10,080 INFO SystemMonitor:191 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-30 08:13:10,081 INFO SystemMonitor:191 [interfaces.py:start():188] Started cpu monitoring 2024-06-30 08:13:10,082 INFO SystemMonitor:191 [interfaces.py:start():188] Started disk monitoring 2024-06-30 08:13:10,082 INFO SystemMonitor:191 [interfaces.py:start():188] Started gpu monitoring 2024-06-30 08:13:10,085 INFO SystemMonitor:191 [interfaces.py:start():188] Started memory monitoring 2024-06-30 08:13:10,086 INFO SystemMonitor:191 [interfaces.py:start():188] Started network monitoring 2024-06-30 08:13:11,031 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:13:11,032 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:13:11,113 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: pause 2024-06-30 08:13:11,113 INFO HandlerThread:191 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-06-30 08:13:11,113 INFO HandlerThread:191 [system_monitor.py:finish():203] Stopping system monitor 2024-06-30 08:13:11,113 DEBUG SystemMonitor:191 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-06-30 08:13:11,113 DEBUG SystemMonitor:191 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-06-30 08:13:11,114 DEBUG SystemMonitor:191 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-06-30 08:13:11,115 INFO HandlerThread:191 [interfaces.py:finish():200] Joined cpu monitor 2024-06-30 08:13:11,115 INFO HandlerThread:191 [interfaces.py:finish():200] Joined disk monitor 2024-06-30 08:13:11,120 INFO HandlerThread:191 [interfaces.py:finish():200] Joined gpu monitor 2024-06-30 08:13:11,121 INFO HandlerThread:191 [interfaces.py:finish():200] Joined memory monitor 2024-06-30 08:13:11,121 INFO HandlerThread:191 [interfaces.py:finish():200] Joined network monitor 2024-06-30 08:13:11,209 DEBUG SenderThread:191 [sender.py:send():378] send: stats 2024-06-30 08:13:11,211 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:13:16,210 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:13:21,211 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:13:26,031 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:13:26,031 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:13:27,164 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:13:32,165 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:13:37,166 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:13:41,031 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:13:41,032 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:13:42,184 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:13:47,185 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:13:52,186 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:13:56,031 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:13:56,032 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:13:57,266 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:14:02,268 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:14:05,705 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: resume 2024-06-30 08:14:05,705 INFO HandlerThread:191 [handler.py:handle_request_resume():715] starting system metrics thread 2024-06-30 08:14:05,705 INFO HandlerThread:191 [system_monitor.py:start():194] Starting system monitor 2024-06-30 08:14:05,705 INFO SystemMonitor:191 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-06-30 08:14:05,706 INFO SystemMonitor:191 [interfaces.py:start():188] Started cpu monitoring 2024-06-30 08:14:05,706 INFO SystemMonitor:191 [interfaces.py:start():188] Started disk monitoring 2024-06-30 08:14:05,707 INFO SystemMonitor:191 [interfaces.py:start():188] Started gpu monitoring 2024-06-30 08:14:05,708 INFO SystemMonitor:191 [interfaces.py:start():188] Started memory monitoring 2024-06-30 08:14:05,709 INFO SystemMonitor:191 [interfaces.py:start():188] Started network monitoring 2024-06-30 08:14:07,269 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:14:11,031 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:14:11,032 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:14:13,179 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:14:18,181 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:14:23,182 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:14:26,032 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: stop_status 2024-06-30 08:14:26,032 DEBUG SenderThread:191 [sender.py:send_request():405] send_request: stop_status 2024-06-30 08:14:28,255 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:14:33,256 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report 2024-06-30 08:14:38,257 DEBUG HandlerThread:191 [handler.py:handle_request():158] handle_request: status_report