2024-07-21 04:03:03,574 INFO StreamThr :147 [internal.py:wandb_internal():85] W&B internal server running at pid: 147, started at: 2024-07-21 04:03:03.573279 2024-07-21 04:03:03,578 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status 2024-07-21 04:03:04,675 INFO WriterThread:147 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/run-nom9e9qu.wandb 2024-07-21 04:03:04,677 DEBUG SenderThread:147 [sender.py:send():379] send: header 2024-07-21 04:03:04,678 DEBUG SenderThread:147 [sender.py:send():379] send: run 2024-07-21 04:03:05,275 INFO SenderThread:147 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/files 2024-07-21 04:03:05,275 INFO SenderThread:147 [sender.py:_start_run_threads():1188] run started: nom9e9qu with start time 1721534583.574145 2024-07-21 04:03:05,285 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: check_version 2024-07-21 04:03:05,286 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: check_version 2024-07-21 04:03:05,370 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: run_start 2024-07-21 04:03:05,384 DEBUG HandlerThread:147 [system_info.py:__init__():26] System info init 2024-07-21 04:03:05,384 DEBUG HandlerThread:147 [system_info.py:__init__():41] System info init done 2024-07-21 04:03:05,384 INFO HandlerThread:147 [system_monitor.py:start():194] Starting system monitor 2024-07-21 04:03:05,384 INFO SystemMonitor:147 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-21 04:03:05,384 INFO HandlerThread:147 [system_monitor.py:probe():214] Collecting system info 2024-07-21 04:03:05,385 INFO SystemMonitor:147 [interfaces.py:start():188] Started cpu monitoring 2024-07-21 04:03:05,386 INFO SystemMonitor:147 [interfaces.py:start():188] Started disk monitoring 2024-07-21 04:03:05,387 INFO SystemMonitor:147 [interfaces.py:start():188] Started gpu monitoring 2024-07-21 04:03:05,388 INFO SystemMonitor:147 [interfaces.py:start():188] Started memory monitoring 2024-07-21 04:03:05,388 INFO SystemMonitor:147 [interfaces.py:start():188] Started network monitoring 2024-07-21 04:03:05,396 DEBUG HandlerThread:147 [system_info.py:probe():152] Probing system 2024-07-21 04:03:05,398 DEBUG HandlerThread:147 [gitlib.py:_init_repo():56] git repository is invalid 2024-07-21 04:03:05,398 DEBUG HandlerThread:147 [system_info.py:probe():200] Probing system done 2024-07-21 04:03:05,398 DEBUG HandlerThread:147 [system_monitor.py:probe():223] {'os': 'Linux-5.15.154+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-07-21T04:03:05.396379', 'startedAt': '2024-07-21T04:03:03.563980', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'f83881ca6137', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.182, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.182, 'min': 0.0, 'max': 0.0}, {'current': 2000.182, 'min': 0.0, 'max': 0.0}, {'current': 2000.182, 'min': 0.0, 'max': 0.0}, {'current': 2000.182, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5777.496520996094}}, 'gpu': 'Tesla P100-PCIE-16GB', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla P100-PCIE-16GB', 'memory_total': 17179869184}], 'memory': {'total': 31.357574462890625}} 2024-07-21 04:03:05,398 INFO HandlerThread:147 [system_monitor.py:probe():224] Finished collecting system info 2024-07-21 04:03:05,398 INFO HandlerThread:147 [system_monitor.py:probe():227] Publishing system info 2024-07-21 04:03:05,398 DEBUG HandlerThread:147 [system_info.py:_save_conda():209] Saving list of conda packages installed into the current environment 2024-07-21 04:03:06,277 INFO Thread-12 :147 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/files/conda-environment.yaml 2024-07-21 04:03:20,412 ERROR HandlerThread:147 [system_info.py:_save_conda():223] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 216, in _save_conda subprocess.call( File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call return p.wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait return self._wait(timeout=timeout) File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait raise TimeoutExpired(self.args, timeout) subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds 2024-07-21 04:03:20,415 DEBUG HandlerThread:147 [system_info.py:_save_conda():224] Saving conda packages done 2024-07-21 04:03:20,416 INFO HandlerThread:147 [system_monitor.py:probe():229] Finished publishing system info 2024-07-21 04:03:20,424 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:03:20,425 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: keepalive 2024-07-21 04:03:20,425 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:03:20,425 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: keepalive 2024-07-21 04:03:20,425 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:03:20,425 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: keepalive 2024-07-21 04:03:20,426 DEBUG SenderThread:147 [sender.py:send():379] send: files 2024-07-21 04:03:20,426 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2024-07-21 04:03:20,999 INFO wandb-upload_0:147 [upload_job.py:push():130] Uploaded file /tmp/tmp2e6z9q77wandb/eoa0x0rf-wandb-metadata.json 2024-07-21 04:03:21,280 INFO Thread-12 :147 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/files/wandb-metadata.json 2024-07-21 04:03:21,398 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: python_packages 2024-07-21 04:03:21,399 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: python_packages 2024-07-21 04:03:21,401 DEBUG SenderThread:147 [sender.py:send():379] send: telemetry 2024-07-21 04:03:21,402 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: stop_status 2024-07-21 04:03:21,402 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:21,403 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: stop_status 2024-07-21 04:03:21,408 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: pause 2024-07-21 04:03:21,408 INFO HandlerThread:147 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-21 04:03:21,408 INFO HandlerThread:147 [system_monitor.py:finish():203] Stopping system monitor 2024-07-21 04:03:21,408 DEBUG SystemMonitor:147 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-21 04:03:21,409 DEBUG SystemMonitor:147 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-21 04:03:21,409 DEBUG SystemMonitor:147 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-21 04:03:21,409 INFO HandlerThread:147 [interfaces.py:finish():200] Joined cpu monitor 2024-07-21 04:03:21,409 INFO HandlerThread:147 [interfaces.py:finish():200] Joined disk monitor 2024-07-21 04:03:21,416 INFO HandlerThread:147 [interfaces.py:finish():200] Joined gpu monitor 2024-07-21 04:03:21,417 INFO HandlerThread:147 [interfaces.py:finish():200] Joined memory monitor 2024-07-21 04:03:21,417 INFO HandlerThread:147 [interfaces.py:finish():200] Joined network monitor 2024-07-21 04:03:21,449 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: resume 2024-07-21 04:03:21,450 INFO HandlerThread:147 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-21 04:03:21,450 INFO HandlerThread:147 [system_monitor.py:start():194] Starting system monitor 2024-07-21 04:03:21,450 INFO SystemMonitor:147 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-21 04:03:21,450 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: pause 2024-07-21 04:03:21,451 INFO SystemMonitor:147 [interfaces.py:start():188] Started cpu monitoring 2024-07-21 04:03:21,451 INFO HandlerThread:147 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-21 04:03:21,451 INFO HandlerThread:147 [system_monitor.py:finish():203] Stopping system monitor 2024-07-21 04:03:21,451 INFO SystemMonitor:147 [interfaces.py:start():188] Started disk monitoring 2024-07-21 04:03:21,452 DEBUG SystemMonitor:147 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-21 04:03:21,452 DEBUG SystemMonitor:147 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-21 04:03:21,452 DEBUG SystemMonitor:147 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-21 04:03:21,453 INFO HandlerThread:147 [interfaces.py:finish():200] Joined cpu monitor 2024-07-21 04:03:21,454 INFO HandlerThread:147 [interfaces.py:finish():200] Joined disk monitor 2024-07-21 04:03:21,455 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: resume 2024-07-21 04:03:21,455 INFO HandlerThread:147 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-21 04:03:21,455 INFO HandlerThread:147 [system_monitor.py:start():194] Starting system monitor 2024-07-21 04:03:21,455 INFO SystemMonitor:147 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-21 04:03:21,455 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: pause 2024-07-21 04:03:21,455 INFO HandlerThread:147 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-21 04:03:21,455 INFO HandlerThread:147 [system_monitor.py:finish():203] Stopping system monitor 2024-07-21 04:03:21,455 WARNING HandlerThread:147 [interfaces.py:finish():205] Failed to finish cpu monitoring: cannot join thread before it is started 2024-07-21 04:03:21,455 INFO SystemMonitor:147 [interfaces.py:start():188] Started cpu monitoring 2024-07-21 04:03:21,456 DEBUG SystemMonitor:147 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-21 04:03:21,456 DEBUG SystemMonitor:147 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-21 04:03:21,456 DEBUG SystemMonitor:147 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-21 04:03:21,456 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: resume 2024-07-21 04:03:21,457 INFO HandlerThread:147 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-21 04:03:21,457 INFO HandlerThread:147 [system_monitor.py:start():194] Starting system monitor 2024-07-21 04:03:21,457 INFO SystemMonitor:147 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-21 04:03:21,457 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: pause 2024-07-21 04:03:21,458 INFO HandlerThread:147 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-21 04:03:21,458 INFO HandlerThread:147 [system_monitor.py:finish():203] Stopping system monitor 2024-07-21 04:03:21,458 WARNING HandlerThread:147 [interfaces.py:finish():205] Failed to finish cpu monitoring: cannot join thread before it is started 2024-07-21 04:03:21,458 INFO SystemMonitor:147 [interfaces.py:start():188] Started cpu monitoring 2024-07-21 04:03:21,458 DEBUG SystemMonitor:147 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-21 04:03:21,458 DEBUG SystemMonitor:147 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-21 04:03:21,458 DEBUG SystemMonitor:147 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-21 04:03:21,467 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: resume 2024-07-21 04:03:21,467 INFO HandlerThread:147 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-21 04:03:21,467 INFO HandlerThread:147 [system_monitor.py:start():194] Starting system monitor 2024-07-21 04:03:21,467 INFO SystemMonitor:147 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-21 04:03:21,468 INFO SystemMonitor:147 [interfaces.py:start():188] Started cpu monitoring 2024-07-21 04:03:21,468 INFO SystemMonitor:147 [interfaces.py:start():188] Started disk monitoring 2024-07-21 04:03:21,468 INFO SystemMonitor:147 [interfaces.py:start():188] Started gpu monitoring 2024-07-21 04:03:21,469 INFO SystemMonitor:147 [interfaces.py:start():188] Started memory monitoring 2024-07-21 04:03:21,470 INFO SystemMonitor:147 [interfaces.py:start():188] Started network monitoring 2024-07-21 04:03:21,471 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: pause 2024-07-21 04:03:21,472 INFO HandlerThread:147 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-21 04:03:21,472 INFO HandlerThread:147 [system_monitor.py:finish():203] Stopping system monitor 2024-07-21 04:03:21,472 DEBUG SystemMonitor:147 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-21 04:03:21,472 DEBUG SystemMonitor:147 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-21 04:03:21,472 DEBUG SystemMonitor:147 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-21 04:03:21,474 INFO HandlerThread:147 [interfaces.py:finish():200] Joined cpu monitor 2024-07-21 04:03:21,474 INFO HandlerThread:147 [interfaces.py:finish():200] Joined disk monitor 2024-07-21 04:03:21,480 INFO HandlerThread:147 [interfaces.py:finish():200] Joined gpu monitor 2024-07-21 04:03:21,481 INFO HandlerThread:147 [interfaces.py:finish():200] Joined memory monitor 2024-07-21 04:03:21,481 INFO HandlerThread:147 [interfaces.py:finish():200] Joined network monitor 2024-07-21 04:03:21,481 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: resume 2024-07-21 04:03:21,481 INFO HandlerThread:147 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-21 04:03:21,481 INFO HandlerThread:147 [system_monitor.py:start():194] Starting system monitor 2024-07-21 04:03:21,482 INFO SystemMonitor:147 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-21 04:03:21,482 INFO SystemMonitor:147 [interfaces.py:start():188] Started cpu monitoring 2024-07-21 04:03:21,482 INFO SystemMonitor:147 [interfaces.py:start():188] Started disk monitoring 2024-07-21 04:03:21,483 INFO SystemMonitor:147 [interfaces.py:start():188] Started gpu monitoring 2024-07-21 04:03:21,484 INFO SystemMonitor:147 [interfaces.py:start():188] Started memory monitoring 2024-07-21 04:03:21,486 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: pause 2024-07-21 04:03:21,486 INFO HandlerThread:147 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-21 04:03:21,486 INFO HandlerThread:147 [system_monitor.py:finish():203] Stopping system monitor 2024-07-21 04:03:21,491 INFO HandlerThread:147 [interfaces.py:finish():200] Joined cpu monitor 2024-07-21 04:03:21,491 INFO SystemMonitor:147 [interfaces.py:start():188] Started network monitoring 2024-07-21 04:03:21,491 DEBUG SystemMonitor:147 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-21 04:03:21,491 DEBUG SystemMonitor:147 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-21 04:03:21,491 DEBUG SystemMonitor:147 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-21 04:03:21,495 INFO HandlerThread:147 [interfaces.py:finish():200] Joined disk monitor 2024-07-21 04:03:21,501 INFO HandlerThread:147 [interfaces.py:finish():200] Joined gpu monitor 2024-07-21 04:03:21,502 INFO HandlerThread:147 [interfaces.py:finish():200] Joined memory monitor 2024-07-21 04:03:21,502 INFO HandlerThread:147 [interfaces.py:finish():200] Joined network monitor 2024-07-21 04:03:21,502 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: resume 2024-07-21 04:03:21,502 INFO HandlerThread:147 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-21 04:03:21,502 INFO HandlerThread:147 [system_monitor.py:start():194] Starting system monitor 2024-07-21 04:03:21,502 INFO SystemMonitor:147 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-21 04:03:21,503 INFO SystemMonitor:147 [interfaces.py:start():188] Started cpu monitoring 2024-07-21 04:03:21,503 INFO SystemMonitor:147 [interfaces.py:start():188] Started disk monitoring 2024-07-21 04:03:21,504 INFO SystemMonitor:147 [interfaces.py:start():188] Started gpu monitoring 2024-07-21 04:03:21,505 INFO SystemMonitor:147 [interfaces.py:start():188] Started memory monitoring 2024-07-21 04:03:21,506 INFO SystemMonitor:147 [interfaces.py:start():188] Started network monitoring 2024-07-21 04:03:21,542 DEBUG SenderThread:147 [sender.py:send():379] send: stats 2024-07-21 04:03:21,543 DEBUG SenderThread:147 [sender.py:send():379] send: stats 2024-07-21 04:03:21,543 DEBUG SenderThread:147 [sender.py:send():379] send: stats 2024-07-21 04:03:21,543 DEBUG SenderThread:147 [sender.py:send():379] send: stats 2024-07-21 04:03:22,005 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: pause 2024-07-21 04:03:22,005 INFO HandlerThread:147 [handler.py:handle_request_pause():724] stopping system metrics thread 2024-07-21 04:03:22,005 INFO HandlerThread:147 [system_monitor.py:finish():203] Stopping system monitor 2024-07-21 04:03:22,005 DEBUG SystemMonitor:147 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-21 04:03:22,005 INFO HandlerThread:147 [interfaces.py:finish():200] Joined cpu monitor 2024-07-21 04:03:22,006 DEBUG SystemMonitor:147 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-21 04:03:22,006 INFO HandlerThread:147 [interfaces.py:finish():200] Joined disk monitor 2024-07-21 04:03:22,006 DEBUG SystemMonitor:147 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-21 04:03:22,014 INFO HandlerThread:147 [interfaces.py:finish():200] Joined gpu monitor 2024-07-21 04:03:22,014 INFO HandlerThread:147 [interfaces.py:finish():200] Joined memory monitor 2024-07-21 04:03:22,014 INFO HandlerThread:147 [interfaces.py:finish():200] Joined network monitor 2024-07-21 04:03:22,014 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: resume 2024-07-21 04:03:22,014 INFO HandlerThread:147 [handler.py:handle_request_resume():715] starting system metrics thread 2024-07-21 04:03:22,015 INFO HandlerThread:147 [system_monitor.py:start():194] Starting system monitor 2024-07-21 04:03:22,015 DEBUG SenderThread:147 [sender.py:send():379] send: stats 2024-07-21 04:03:22,016 INFO SystemMonitor:147 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-21 04:03:22,016 INFO SystemMonitor:147 [interfaces.py:start():188] Started cpu monitoring 2024-07-21 04:03:22,017 INFO SystemMonitor:147 [interfaces.py:start():188] Started disk monitoring 2024-07-21 04:03:22,017 INFO SystemMonitor:147 [interfaces.py:start():188] Started gpu monitoring 2024-07-21 04:03:22,018 INFO SystemMonitor:147 [interfaces.py:start():188] Started memory monitoring 2024-07-21 04:03:22,019 INFO SystemMonitor:147 [interfaces.py:start():188] Started network monitoring 2024-07-21 04:03:22,281 INFO Thread-12 :147 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/files/output.log 2024-07-21 04:03:22,281 INFO Thread-12 :147 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/files/requirements.txt 2024-07-21 04:03:22,400 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:22,584 DEBUG SenderThread:147 [sender.py:send():379] send: config 2024-07-21 04:03:22,585 DEBUG SenderThread:147 [sender.py:send():379] send: telemetry 2024-07-21 04:03:22,585 DEBUG SenderThread:147 [sender.py:send():379] send: metric 2024-07-21 04:03:22,585 DEBUG SenderThread:147 [sender.py:send():379] send: telemetry 2024-07-21 04:03:22,586 DEBUG SenderThread:147 [sender.py:send():379] send: metric 2024-07-21 04:03:22,586 WARNING SenderThread:147 [sender.py:send_metric():1405] Seen metric with glob (shouldn't happen) 2024-07-21 04:03:22,586 DEBUG SenderThread:147 [sender.py:send():379] send: telemetry 2024-07-21 04:03:22,799 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: log_artifact 2024-07-21 04:03:22,799 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: log_artifact 2024-07-21 04:03:22,959 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: log_artifact 2024-07-21 04:03:23,400 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:24,282 INFO Thread-12 :147 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/files/output.log 2024-07-21 04:03:24,401 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:24,759 INFO wandb-upload_0:147 [upload_job.py:push():88] Uploaded file /tmp/tmpr9hsr9h9/model_architecture.txt 2024-07-21 04:03:25,401 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:25,422 INFO SenderThread:147 [sender.py:send_request_log_artifact():1518] logged artifact model-nom9e9qu - {'id': 'QXJ0aWZhY3Q6MTAxMDgzNDA2MQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjI5OTExOTI3MA==', 'latestArtifact': None}} 2024-07-21 04:03:25,422 DEBUG SenderThread:147 [sender.py:send():379] send: config 2024-07-21 04:03:25,423 DEBUG SenderThread:147 [sender.py:send():379] send: metric 2024-07-21 04:03:25,423 DEBUG SenderThread:147 [sender.py:send():379] send: metric 2024-07-21 04:03:25,424 WARNING SenderThread:147 [sender.py:send_metric():1405] Seen metric with glob (shouldn't happen) 2024-07-21 04:03:25,424 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: log_artifact 2024-07-21 04:03:25,429 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:03:25,609 INFO SenderThread:147 [sender.py:send_request_log_artifact():1518] logged artifact model-nom9e9qu - {'id': 'QXJ0aWZhY3Q6MTAxMDgzNDA2MQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjI5OTExOTI3MA==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTAxMDgzNDA2MQ==', 'versionIndex': 0}}} 2024-07-21 04:03:26,401 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:27,401 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:28,401 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:28,612 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: partial_history 2024-07-21 04:03:28,615 DEBUG SenderThread:147 [sender.py:send():379] send: metric 2024-07-21 04:03:28,615 DEBUG SenderThread:147 [sender.py:send():379] send: metric 2024-07-21 04:03:28,616 DEBUG SenderThread:147 [sender.py:send():379] send: metric 2024-07-21 04:03:28,616 DEBUG SenderThread:147 [sender.py:send():379] send: metric 2024-07-21 04:03:28,616 DEBUG SenderThread:147 [sender.py:send():379] send: history 2024-07-21 04:03:28,616 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:03:28,618 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:03:28,653 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: partial_history 2024-07-21 04:03:28,655 DEBUG SenderThread:147 [sender.py:send():379] send: history 2024-07-21 04:03:28,655 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:03:28,655 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:03:29,284 INFO Thread-12 :147 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/files/wandb-summary.json 2024-07-21 04:03:29,401 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:30,401 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:30,656 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:03:31,401 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:31,538 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: partial_history 2024-07-21 04:03:31,539 DEBUG SenderThread:147 [sender.py:send():379] send: history 2024-07-21 04:03:31,540 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:03:31,540 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:03:31,578 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: partial_history 2024-07-21 04:03:31,579 DEBUG SenderThread:147 [sender.py:send():379] send: history 2024-07-21 04:03:31,580 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:03:31,580 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:03:32,285 INFO Thread-12 :147 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/files/wandb-summary.json 2024-07-21 04:03:32,402 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:33,402 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:34,402 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:34,414 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: partial_history 2024-07-21 04:03:34,415 DEBUG SenderThread:147 [sender.py:send():379] send: history 2024-07-21 04:03:34,415 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:03:34,415 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:03:34,454 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: partial_history 2024-07-21 04:03:34,455 DEBUG SenderThread:147 [sender.py:send():379] send: history 2024-07-21 04:03:34,456 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:03:34,456 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:03:35,286 INFO Thread-12 :147 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/files/wandb-summary.json 2024-07-21 04:03:35,402 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:36,399 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: stop_status 2024-07-21 04:03:36,400 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: stop_status 2024-07-21 04:03:36,440 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:36,653 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:03:37,182 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: partial_history 2024-07-21 04:03:37,183 DEBUG SenderThread:147 [sender.py:send():379] send: history 2024-07-21 04:03:37,184 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:03:37,184 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:03:37,222 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: partial_history 2024-07-21 04:03:37,224 DEBUG SenderThread:147 [sender.py:send():379] send: history 2024-07-21 04:03:37,224 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:03:37,224 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:03:37,287 INFO Thread-12 :147 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/files/wandb-summary.json 2024-07-21 04:03:37,288 INFO Thread-12 :147 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/files/config.yaml 2024-07-21 04:03:37,402 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:38,403 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:38,927 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: partial_history 2024-07-21 04:03:38,928 DEBUG SenderThread:147 [sender.py:send():379] send: history 2024-07-21 04:03:38,929 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:03:38,930 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:03:38,967 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: partial_history 2024-07-21 04:03:38,968 DEBUG SenderThread:147 [sender.py:send():379] send: history 2024-07-21 04:03:38,968 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:03:38,969 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:03:39,288 INFO Thread-12 :147 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/files/wandb-summary.json 2024-07-21 04:03:39,403 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:40,653 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:41,653 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:42,401 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:03:42,782 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:43,782 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:45,630 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:47,402 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:03:50,476 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:52,477 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:03:53,067 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:53,134 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: stop_status 2024-07-21 04:03:53,135 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: stop_status 2024-07-21 04:03:55,898 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:56,935 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:57,626 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:57,792 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:03:58,626 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:03:59,626 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:00,627 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:01,627 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:02,627 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:03,598 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:04:03,627 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:04,627 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:05,628 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:06,411 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: stop_status 2024-07-21 04:04:06,411 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: stop_status 2024-07-21 04:04:06,628 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:07,628 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:08,628 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:09,581 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:04:09,629 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:10,629 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:11,629 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:12,631 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:13,641 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:14,598 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:04:14,635 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:15,310 DEBUG SenderThread:147 [sender.py:send():379] send: telemetry 2024-07-21 04:04:15,351 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: partial_history 2024-07-21 04:04:15,353 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: partial_history 2024-07-21 04:04:15,354 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:04:15,636 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:16,304 INFO Thread-12 :147 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/files/wandb-summary.json 2024-07-21 04:04:16,636 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:17,636 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:18,636 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:19,636 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:20,636 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:21,411 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: stop_status 2024-07-21 04:04:21,636 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:22,019 DEBUG SystemMonitor:147 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-21 04:04:22,636 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:23,637 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:24,637 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:25,637 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:26,637 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:27,637 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:28,638 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:29,638 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:30,638 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:31,638 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:32,638 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:33,638 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:34,639 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:35,130 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:04:35,130 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:04:35,131 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:04:35,134 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:04:35,134 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:04:35,166 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:04:35,167 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:04:35,168 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:04:35,168 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:04:35,312 INFO Thread-12 :147 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/files/wandb-summary.json 2024-07-21 04:04:35,332 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:04:35,333 DEBUG SenderThread:147 [sender.py:send():379] send: history 2024-07-21 04:04:35,333 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:04:35,334 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:04:35,334 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:04:35,366 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:04:35,366 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:04:35,367 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:04:35,367 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:04:35,370 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:04:35,371 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:04:35,399 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:04:35,399 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:04:35,400 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:04:35,400 DEBUG SenderThread:147 [sender.py:send():379] send: history 2024-07-21 04:04:35,401 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: summary_record 2024-07-21 04:04:35,404 INFO SenderThread:147 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-21 04:04:35,405 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: stop_status 2024-07-21 04:04:35,552 DEBUG SenderThread:147 [sender.py:send():379] send: stats 2024-07-21 04:04:35,639 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:36,312 INFO Thread-12 :147 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/files/wandb-summary.json 2024-07-21 04:04:36,411 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: stop_status 2024-07-21 04:04:36,412 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: stop_status 2024-07-21 04:04:36,639 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:37,639 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:38,639 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:39,640 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:40,640 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:40,652 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:04:41,314 INFO Thread-12 :147 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240721_040303-nom9e9qu/files/config.yaml 2024-07-21 04:04:41,640 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:42,640 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:43,640 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:44,641 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:45,641 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:45,874 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:04:46,641 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:47,641 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:48,641 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:49,642 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:50,642 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:50,875 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:04:51,411 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: stop_status 2024-07-21 04:04:51,412 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: stop_status 2024-07-21 04:04:51,642 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:52,021 DEBUG SenderThread:147 [sender.py:send():379] send: stats 2024-07-21 04:04:52,642 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:53,642 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:54,643 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:55,643 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:56,022 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:04:56,643 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:57,643 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:58,643 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:04:59,643 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:00,643 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:01,023 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:05:01,644 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:02,644 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:03,644 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:04,644 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:05,644 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:06,025 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:05:06,412 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: stop_status 2024-07-21 04:05:06,412 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: stop_status 2024-07-21 04:05:06,644 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:07,645 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:08,645 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:09,645 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:10,645 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:11,560 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:05:11,645 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:12,646 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:13,646 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:14,646 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:15,646 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:16,561 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:05:16,646 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:17,647 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:18,647 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:19,647 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:20,647 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:21,412 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: stop_status 2024-07-21 04:05:21,412 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: stop_status 2024-07-21 04:05:21,648 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:21,649 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:05:22,022 DEBUG SenderThread:147 [sender.py:send():379] send: stats 2024-07-21 04:05:22,648 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:23,648 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:24,648 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:25,649 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:26,649 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:27,023 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:05:27,649 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:28,649 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:29,649 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:30,649 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:31,650 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:32,025 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:05:32,650 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:33,650 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:34,650 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:35,650 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:36,412 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: stop_status 2024-07-21 04:05:36,412 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: stop_status 2024-07-21 04:05:36,650 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:37,651 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:37,655 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:05:38,651 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:39,651 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:40,651 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:41,651 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:42,652 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:42,656 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:05:43,652 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:44,652 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:45,652 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:46,652 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:47,652 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:47,657 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:05:48,652 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:49,653 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:50,653 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:51,412 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: stop_status 2024-07-21 04:05:51,413 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: stop_status 2024-07-21 04:05:51,653 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:52,023 DEBUG SenderThread:147 [sender.py:send():379] send: stats 2024-07-21 04:05:52,653 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:53,023 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:05:53,654 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:54,654 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:55,654 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:56,654 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:57,654 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:58,024 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:05:58,655 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:05:59,655 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:00,655 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:01,655 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:02,656 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:03,026 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:06:03,657 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:04,656 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:05,656 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:06,413 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: stop_status 2024-07-21 04:06:06,413 DEBUG SenderThread:147 [sender.py:send_request():406] send_request: stop_status 2024-07-21 04:06:06,656 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:07,656 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:08,565 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:06:08,656 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:09,657 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:10,657 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:11,657 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:12,657 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:13,566 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: status_report 2024-07-21 04:06:13,657 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:14,657 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:15,658 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-21 04:06:16,658 DEBUG HandlerThread:147 [handler.py:handle_request():158] handle_request: internal_messages