2024-07-20 09:10:16,700 INFO    StreamThr :107 [internal.py:wandb_internal():85] W&B internal server running at pid: 107, started at: 2024-07-20 09:10:16.699473
2024-07-20 09:10:16,702 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status
2024-07-20 09:10:16,710 INFO    WriterThread:107 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/run-cu9n4dsx.wandb
2024-07-20 09:10:16,711 DEBUG   SenderThread:107 [sender.py:send():379] send: header
2024-07-20 09:10:16,713 DEBUG   SenderThread:107 [sender.py:send():379] send: run
2024-07-20 09:10:17,727 INFO    SenderThread:107 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files
2024-07-20 09:10:17,727 INFO    SenderThread:107 [sender.py:_start_run_threads():1188] run started: cu9n4dsx with start time 1721466616.70067
2024-07-20 09:10:17,740 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: check_version
2024-07-20 09:10:17,741 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: check_version
2024-07-20 09:10:17,799 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: run_start
2024-07-20 09:10:17,812 DEBUG   HandlerThread:107 [system_info.py:__init__():26] System info init
2024-07-20 09:10:17,812 DEBUG   HandlerThread:107 [system_info.py:__init__():41] System info init done
2024-07-20 09:10:17,812 INFO    HandlerThread:107 [system_monitor.py:start():194] Starting system monitor
2024-07-20 09:10:17,813 INFO    SystemMonitor:107 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-07-20 09:10:17,813 INFO    HandlerThread:107 [system_monitor.py:probe():214] Collecting system info
2024-07-20 09:10:17,813 INFO    SystemMonitor:107 [interfaces.py:start():188] Started cpu monitoring
2024-07-20 09:10:17,814 INFO    SystemMonitor:107 [interfaces.py:start():188] Started disk monitoring
2024-07-20 09:10:17,815 INFO    SystemMonitor:107 [interfaces.py:start():188] Started gpu monitoring
2024-07-20 09:10:17,816 INFO    SystemMonitor:107 [interfaces.py:start():188] Started memory monitoring
2024-07-20 09:10:17,818 INFO    SystemMonitor:107 [interfaces.py:start():188] Started network monitoring
2024-07-20 09:10:17,823 DEBUG   HandlerThread:107 [system_info.py:probe():152] Probing system
2024-07-20 09:10:17,826 DEBUG   HandlerThread:107 [gitlib.py:_init_repo():56] git repository is invalid
2024-07-20 09:10:17,826 DEBUG   HandlerThread:107 [system_info.py:probe():200] Probing system done
2024-07-20 09:10:17,826 DEBUG   HandlerThread:107 [system_monitor.py:probe():223] {'os': 'Linux-5.15.154+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-07-20T09:10:17.823430', 'startedAt': '2024-07-20T09:10:16.692039', 'docker': None, 'cuda': None, 'args': ('-f', '/tmp/tmpiwb4ut_j.json', '--HistoryManager.hist_file=:memory:'), 'state': 'running', 'program': '<python with no main file>', 'codePathLocal': None, 'host': '983c1ca165f0', 'username': 'root', 'executable': '/opt/conda/bin/python', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.176, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.176, 'min': 0.0, 'max': 0.0}, {'current': 2000.176, 'min': 0.0, 'max': 0.0}, {'current': 2000.176, 'min': 0.0, 'max': 0.0}, {'current': 2000.176, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5777.478328704834}}, 'gpu': 'Tesla P100-PCIE-16GB', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla P100-PCIE-16GB', 'memory_total': 17179869184}], 'memory': {'total': 31.357574462890625}}
2024-07-20 09:10:17,826 INFO    HandlerThread:107 [system_monitor.py:probe():224] Finished collecting system info
2024-07-20 09:10:17,826 INFO    HandlerThread:107 [system_monitor.py:probe():227] Publishing system info
2024-07-20 09:10:17,826 DEBUG   HandlerThread:107 [system_info.py:_save_conda():209] Saving list of conda packages installed into the current environment
2024-07-20 09:10:18,729 INFO    Thread-12 :107 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/conda-environment.yaml
2024-07-20 09:10:32,842 ERROR   HandlerThread:107 [system_info.py:_save_conda():223] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 216, in _save_conda
    subprocess.call(
  File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call
    return p.wait(timeout=timeout)
  File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait
    return self._wait(timeout=timeout)
  File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait
    raise TimeoutExpired(self.args, timeout)
subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds
2024-07-20 09:10:32,844 DEBUG   HandlerThread:107 [system_info.py:_save_conda():224] Saving conda packages done
2024-07-20 09:10:32,845 INFO    HandlerThread:107 [system_monitor.py:probe():229] Finished publishing system info
2024-07-20 09:10:32,854 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:10:32,854 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:10:32,854 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:10:32,855 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:10:32,855 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:10:32,855 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:10:32,855 DEBUG   SenderThread:107 [sender.py:send():379] send: files
2024-07-20 09:10:32,855 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now
2024-07-20 09:10:33,138 INFO    wandb-upload_0:107 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_94reawandb/3fq0d7v4-wandb-metadata.json
2024-07-20 09:10:33,733 INFO    Thread-12 :107 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/wandb-metadata.json
2024-07-20 09:10:33,836 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: python_packages
2024-07-20 09:10:33,837 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: python_packages
2024-07-20 09:10:33,839 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: stop_status
2024-07-20 09:10:33,840 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: stop_status
2024-07-20 09:10:33,842 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:33,984 DEBUG   SenderThread:107 [sender.py:send():379] send: telemetry
2024-07-20 09:10:34,733 INFO    Thread-12 :107 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/requirements.txt
2024-07-20 09:10:34,838 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:35,734 INFO    Thread-12 :107 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/output.log
2024-07-20 09:10:36,170 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:36,366 DEBUG   SenderThread:107 [sender.py:send():379] send: config
2024-07-20 09:10:36,367 DEBUG   SenderThread:107 [sender.py:send():379] send: telemetry
2024-07-20 09:10:36,368 DEBUG   SenderThread:107 [sender.py:send():379] send: metric
2024-07-20 09:10:36,368 DEBUG   SenderThread:107 [sender.py:send():379] send: telemetry
2024-07-20 09:10:36,368 DEBUG   SenderThread:107 [sender.py:send():379] send: metric
2024-07-20 09:10:36,368 WARNING SenderThread:107 [sender.py:send_metric():1405] Seen metric with glob (shouldn't happen)
2024-07-20 09:10:36,368 DEBUG   SenderThread:107 [sender.py:send():379] send: telemetry
2024-07-20 09:10:36,511 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: log_artifact
2024-07-20 09:10:36,511 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: log_artifact
2024-07-20 09:10:36,605 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: log_artifact
2024-07-20 09:10:37,155 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:37,240 INFO    wandb-upload_0:107 [upload_job.py:push():88] Uploaded file /tmp/tmpdd7y_ppq/model_architecture.txt
2024-07-20 09:10:37,696 INFO    SenderThread:107 [sender.py:send_request_log_artifact():1518] logged artifact model-cu9n4dsx - {'id': 'QXJ0aWZhY3Q6MTAwNjI0ODQ2Mg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjI5NDk2NDQ3Mw==', 'latestArtifact': None}}
2024-07-20 09:10:37,696 DEBUG   SenderThread:107 [sender.py:send():379] send: config
2024-07-20 09:10:37,697 DEBUG   SenderThread:107 [sender.py:send():379] send: metric
2024-07-20 09:10:37,697 DEBUG   SenderThread:107 [sender.py:send():379] send: metric
2024-07-20 09:10:37,697 WARNING SenderThread:107 [sender.py:send_metric():1405] Seen metric with glob (shouldn't happen)
2024-07-20 09:10:37,698 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: log_artifact
2024-07-20 09:10:37,703 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:10:37,740 INFO    Thread-12 :107 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/output.log
2024-07-20 09:10:37,790 INFO    SenderThread:107 [sender.py:send_request_log_artifact():1518] logged artifact model-cu9n4dsx - {'id': 'QXJ0aWZhY3Q6MTAwNjI0ODQ2Mg==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjI5NDk2NDQ3Mw==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTAwNjI0ODQ2Mg==', 'versionIndex': 0}}}
2024-07-20 09:10:38,156 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:39,156 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:40,104 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:40,106 DEBUG   SenderThread:107 [sender.py:send():379] send: metric
2024-07-20 09:10:40,106 DEBUG   SenderThread:107 [sender.py:send():379] send: metric
2024-07-20 09:10:40,106 DEBUG   SenderThread:107 [sender.py:send():379] send: metric
2024-07-20 09:10:40,106 DEBUG   SenderThread:107 [sender.py:send():379] send: metric
2024-07-20 09:10:40,107 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:40,107 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:40,109 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:40,144 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:40,145 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:40,145 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:40,145 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:40,156 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:40,742 INFO    Thread-12 :107 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/wandb-summary.json
2024-07-20 09:10:41,156 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:41,504 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:41,505 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:41,506 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:41,506 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:41,545 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:41,546 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:41,546 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:41,546 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:41,742 INFO    Thread-12 :107 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/wandb-summary.json
2024-07-20 09:10:42,156 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:42,854 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:42,855 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:42,855 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:42,857 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:10:42,858 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:42,895 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:42,896 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:42,896 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:42,896 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:43,157 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:43,743 INFO    Thread-12 :107 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/wandb-summary.json
2024-07-20 09:10:44,157 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:44,307 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:44,309 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:44,309 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:44,310 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:44,348 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:44,348 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:44,349 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:44,349 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:44,743 INFO    Thread-12 :107 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/wandb-summary.json
2024-07-20 09:10:45,157 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:45,682 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:45,683 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:45,683 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:45,686 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:45,723 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:45,724 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:45,724 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:45,724 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:45,743 INFO    Thread-12 :107 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/wandb-summary.json
2024-07-20 09:10:46,158 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:47,157 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:47,194 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:47,195 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:47,195 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:47,196 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:47,235 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:47,235 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:47,236 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:47,236 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:47,744 INFO    Thread-12 :107 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/wandb-summary.json
2024-07-20 09:10:48,158 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:48,242 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:10:48,575 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:48,576 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:48,577 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:48,579 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:48,616 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:48,617 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:48,617 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:48,617 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:48,745 INFO    Thread-12 :107 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/config.yaml
2024-07-20 09:10:48,745 INFO    Thread-12 :107 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/wandb-summary.json
2024-07-20 09:10:48,838 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: stop_status
2024-07-20 09:10:48,838 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: stop_status
2024-07-20 09:10:49,158 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:49,911 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:49,913 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:49,913 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:49,913 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:49,953 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:49,954 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:49,954 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:49,954 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:50,158 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:50,746 INFO    Thread-12 :107 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/wandb-summary.json
2024-07-20 09:10:51,158 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:51,461 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:51,462 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:51,462 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:51,464 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:51,501 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:51,502 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:51,502 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:51,502 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:51,746 INFO    Thread-12 :107 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/wandb-summary.json
2024-07-20 09:10:52,158 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:52,354 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:52,355 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:52,355 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:52,356 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:52,395 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:10:52,396 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:10:52,396 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:10:52,396 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:10:52,746 INFO    Thread-12 :107 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/wandb-summary.json
2024-07-20 09:10:53,159 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:53,397 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:10:54,163 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:55,163 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:56,180 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:57,190 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:58,990 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:10:59,049 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:11:04,050 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:11:04,862 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: stop_status
2024-07-20 09:11:04,863 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: stop_status
2024-07-20 09:11:04,904 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: internal_messages
2024-07-20 09:11:09,986 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:11:10,603 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:11:15,667 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:11:17,818 DEBUG   SystemMonitor:107 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-07-20 09:11:17,820 DEBUG   SenderThread:107 [sender.py:send():379] send: stats
2024-07-20 09:11:19,860 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: stop_status
2024-07-20 09:11:19,860 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: stop_status
2024-07-20 09:11:20,668 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:11:25,668 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:11:27,521 DEBUG   SenderThread:107 [sender.py:send():379] send: telemetry
2024-07-20 09:11:27,563 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:11:27,566 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: partial_history
2024-07-20 09:11:27,567 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:11:27,761 INFO    Thread-12 :107 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/wandb-summary.json
2024-07-20 09:11:34,860 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: stop_status
2024-07-20 09:11:39,861 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:11:44,862 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:11:45,872 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:11:45,873 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:11:45,873 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:11:45,874 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:11:45,874 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:11:45,912 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:11:45,913 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:11:45,913 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:11:45,914 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:11:45,918 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:11:45,918 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:11:45,919 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:11:46,099 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:11:46,100 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:11:46,101 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:11:46,101 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:11:46,146 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:11:46,147 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:11:46,148 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:11:46,148 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:11:46,193 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:11:46,193 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:11:46,194 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:11:46,194 DEBUG   SenderThread:107 [sender.py:send():379] send: history
2024-07-20 09:11:46,194 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: summary_record
2024-07-20 09:11:46,196 INFO    SenderThread:107 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-07-20 09:11:46,196 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: stop_status
2024-07-20 09:11:46,768 INFO    Thread-12 :107 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/wandb-summary.json
2024-07-20 09:11:47,820 DEBUG   SenderThread:107 [sender.py:send():379] send: stats
2024-07-20 09:11:49,860 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: stop_status
2024-07-20 09:11:49,861 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: stop_status
2024-07-20 09:11:50,950 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:11:51,771 INFO    Thread-12 :107 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240720_091016-cu9n4dsx/files/config.yaml
2024-07-20 09:11:55,673 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:11:56,074 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:12:00,674 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:12:01,075 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:12:04,860 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: stop_status
2024-07-20 09:12:04,861 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: stop_status
2024-07-20 09:12:07,014 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:12:10,676 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:12:12,016 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:12:15,678 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:12:17,017 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:12:17,821 DEBUG   SenderThread:107 [sender.py:send():379] send: stats
2024-07-20 09:12:19,860 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: stop_status
2024-07-20 09:12:19,861 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: stop_status
2024-07-20 09:12:22,032 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:12:25,680 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:12:27,034 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:12:30,682 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:12:32,035 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:12:34,861 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: stop_status
2024-07-20 09:12:34,861 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: stop_status
2024-07-20 09:12:37,932 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:12:40,684 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:12:42,933 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:12:45,686 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:12:47,822 DEBUG   SenderThread:107 [sender.py:send():379] send: stats
2024-07-20 09:12:48,823 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:12:49,861 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: stop_status
2024-07-20 09:12:49,861 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: stop_status
2024-07-20 09:12:53,981 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:12:55,688 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:12:58,982 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:13:00,689 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:13:03,983 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:13:04,861 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: stop_status
2024-07-20 09:13:04,861 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: stop_status
2024-07-20 09:13:09,033 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:13:10,691 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:13:14,034 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:13:15,693 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive
2024-07-20 09:13:17,823 DEBUG   SenderThread:107 [sender.py:send():379] send: stats
2024-07-20 09:13:19,824 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:13:19,861 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: stop_status
2024-07-20 09:13:19,861 DEBUG   SenderThread:107 [sender.py:send_request():406] send_request: stop_status
2024-07-20 09:13:24,969 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: status_report
2024-07-20 09:13:25,697 DEBUG   HandlerThread:107 [handler.py:handle_request():158] handle_request: keepalive