| 2024-02-26 09:21:05,531 INFO StreamThr :196 [internal.py:wandb_internal():86] W&B internal server running at pid: 196, started at: 2024-02-26 09:21:05.531199 | |
| 2024-02-26 09:21:05,533 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status | |
| 2024-02-26 09:21:06,208 INFO WriterThread:196 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240226_092105-91uhs91a/run-91uhs91a.wandb | |
| 2024-02-26 09:21:06,208 DEBUG SenderThread:196 [sender.py:send():382] send: header | |
| 2024-02-26 09:21:06,211 DEBUG SenderThread:196 [sender.py:send():382] send: run | |
| 2024-02-26 09:21:06,389 INFO SenderThread:196 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files | |
| 2024-02-26 09:21:06,389 INFO SenderThread:196 [sender.py:_start_run_threads():1136] run started: 91uhs91a with start time 1708939265.532258 | |
| 2024-02-26 09:21:06,397 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-02-26 09:21:06,397 DEBUG SenderThread:196 [sender.py:send_request():409] send_request: check_version | |
| 2024-02-26 09:21:06,464 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-02-26 09:21:06,475 DEBUG HandlerThread:196 [system_info.py:__init__():27] System info init | |
| 2024-02-26 09:21:06,475 DEBUG HandlerThread:196 [system_info.py:__init__():42] System info init done | |
| 2024-02-26 09:21:06,475 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:21:06,475 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:21:06,475 INFO HandlerThread:196 [system_monitor.py:probe():214] Collecting system info | |
| 2024-02-26 09:21:06,476 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:21:06,476 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:21:06,477 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:21:06,477 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:21:06,478 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:21:06,484 DEBUG HandlerThread:196 [system_info.py:probe():151] Probing system | |
| 2024-02-26 09:21:06,486 DEBUG HandlerThread:196 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-02-26 09:21:06,486 DEBUG HandlerThread:196 [system_info.py:probe():199] Probing system done | |
| 2024-02-26 09:21:06,487 DEBUG HandlerThread:196 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-02-26T09:21:06.484258', 'startedAt': '2024-02-26T09:21:05.525040', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': 'd79522c38279', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.186, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.186, 'min': 0.0, 'max': 0.0}, {'current': 2000.186, 'min': 0.0, 'max': 0.0}, {'current': 2000.186, 'min': 0.0, 'max': 0.0}, {'current': 2000.186, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5437.818241119385}}, 'gpu': 'Tesla P100-PCIE-16GB', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla P100-PCIE-16GB', 'memory_total': 17179869184}], 'memory': {'total': 31.357559204101562}} | |
| 2024-02-26 09:21:06,487 INFO HandlerThread:196 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-02-26 09:21:06,487 INFO HandlerThread:196 [system_monitor.py:probe():227] Publishing system info | |
| 2024-02-26 09:21:06,487 DEBUG HandlerThread:196 [system_info.py:_save_conda():208] Saving list of conda packages installed into the current environment | |
| 2024-02-26 09:21:07,391 INFO Thread-12 :196 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/conda-environment.yaml | |
| 2024-02-26 09:21:58,406 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/conda-environment.yaml | |
| 2024-02-26 09:21:58,864 DEBUG HandlerThread:196 [system_info.py:_save_conda():220] Saving conda packages done | |
| 2024-02-26 09:21:58,865 INFO HandlerThread:196 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-02-26 09:21:58,872 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:21:58,872 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:21:58,873 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:21:58,873 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:21:58,873 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:21:58,873 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:21:58,873 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:21:58,873 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:21:58,873 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:21:58,873 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:21:58,873 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:21:58,873 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-02-26 09:21:58,873 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-26 09:21:58,873 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-26 09:21:58,874 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:21:58,874 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:21:58,874 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:21:58,875 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:21:58,875 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:21:58,875 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:21:58,876 DEBUG SenderThread:196 [sender.py:send_request():409] send_request: python_packages | |
| 2024-02-26 09:21:58,878 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:21:58,879 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:21:58,879 DEBUG SenderThread:196 [sender.py:send():382] send: telemetry | |
| 2024-02-26 09:21:58,879 DEBUG SenderThread:196 [sender.py:send():382] send: config | |
| 2024-02-26 09:21:58,881 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:21:58,881 DEBUG SenderThread:196 [sender.py:send():382] send: telemetry | |
| 2024-02-26 09:21:58,881 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:21:58,881 WARNING SenderThread:196 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) | |
| 2024-02-26 09:21:58,881 DEBUG SenderThread:196 [sender.py:send():382] send: telemetry | |
| 2024-02-26 09:21:58,882 DEBUG SenderThread:196 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-26 09:21:58,891 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:21:58,892 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:21:58,892 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:21:58,892 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:21:58,892 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:21:58,892 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:21:58,892 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:21:58,892 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:21:58,892 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:21:58,893 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:21:58,893 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:21:58,893 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:21:58,893 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:21:58,893 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:21:58,893 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:21:58,893 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:21:58,894 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:21:58,895 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:21:58,896 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:21:58,897 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:21:59,035 DEBUG SenderThread:196 [sender.py:send():382] send: config | |
| 2024-02-26 09:21:59,037 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:21:59,037 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:21:59,037 WARNING SenderThread:196 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) | |
| 2024-02-26 09:21:59,037 DEBUG SenderThread:196 [sender.py:send():382] send: files | |
| 2024-02-26 09:21:59,038 INFO SenderThread:196 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now | |
| 2024-02-26 09:21:59,038 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:21:59,335 INFO wandb-upload_0:196 [upload_job.py:push():131] Uploaded file /tmp/tmp5wprt0czwandb/qmdsgk7x-wandb-metadata.json | |
| 2024-02-26 09:21:59,406 INFO Thread-12 :196 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/wandb-metadata.json | |
| 2024-02-26 09:21:59,406 INFO Thread-12 :196 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/requirements.txt | |
| 2024-02-26 09:21:59,602 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:21:59,602 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:21:59,602 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:21:59,603 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:21:59,603 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:21:59,603 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:21:59,604 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:21:59,604 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:21:59,611 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:21:59,611 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:21:59,611 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:21:59,611 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:22:01,613 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:22:01,749 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:22:06,619 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:22:06,755 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:22:07,409 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/config.yaml | |
| 2024-02-26 09:22:08,331 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:22:08,331 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:22:08,331 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:22:08,331 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:22:08,332 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:22:08,332 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:22:08,335 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:22:08,335 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:22:08,336 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:22:09,404 DEBUG SenderThread:196 [sender.py:send():382] send: config | |
| 2024-02-26 09:22:09,406 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:22:09,406 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:22:09,406 WARNING SenderThread:196 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) | |
| 2024-02-26 09:22:09,927 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:22:09,927 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:22:09,927 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:22:09,927 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:22:09,928 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:22:09,928 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:22:09,928 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:22:09,928 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:22:09,934 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:22:09,934 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:22:09,934 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:22:09,935 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:22:11,810 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:22:11,936 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:22:16,811 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:22:16,936 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:22:18,738 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:22:18,738 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:22:18,738 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:22:18,738 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:22:18,739 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:22:18,740 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:22:18,740 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:22:18,742 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:22:18,743 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:22:19,271 DEBUG SenderThread:196 [sender.py:send():382] send: config | |
| 2024-02-26 09:22:19,272 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:22:19,273 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:22:19,273 WARNING SenderThread:196 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) | |
| 2024-02-26 09:22:19,668 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:22:19,669 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:22:19,669 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:22:19,669 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:22:19,669 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:22:19,669 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:22:19,670 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:22:19,670 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:22:19,675 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:22:19,676 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:22:19,676 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:22:19,676 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:22:21,813 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:22:22,677 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:22:26,814 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:22:27,678 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:22:31,815 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:22:32,679 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:22:36,816 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:22:37,685 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:22:37,695 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:22:37,696 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:22:37,696 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:22:37,696 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:22:37,696 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:22:37,697 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:22:37,699 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:22:37,701 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:22:37,702 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:22:38,419 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/config.yaml | |
| 2024-02-26 09:22:38,766 DEBUG SenderThread:196 [sender.py:send():382] send: config | |
| 2024-02-26 09:22:38,768 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:22:38,768 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:22:38,768 WARNING SenderThread:196 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) | |
| 2024-02-26 09:22:42,174 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:22:43,769 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:22:47,175 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:22:48,770 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:22:52,181 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:22:53,770 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:22:57,184 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:22:58,771 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:23:02,185 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:23:03,772 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:23:07,189 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:23:08,778 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:23:09,430 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/config.yaml | |
| 2024-02-26 09:23:12,191 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:23:13,902 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:23:17,192 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:23:18,902 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:23:19,821 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:23:19,821 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:23:19,821 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:23:19,821 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:23:19,821 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:23:19,821 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:23:19,822 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:23:19,822 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:23:19,830 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:23:19,830 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:23:19,830 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:23:19,830 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:23:22,193 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:23:24,832 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:23:27,194 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:23:29,832 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:23:32,195 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:23:32,481 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:23:32,481 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:23:32,481 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:23:32,482 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:23:32,482 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:23:32,483 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:23:32,484 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:23:32,485 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:23:32,486 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:23:33,088 DEBUG SenderThread:196 [sender.py:send():382] send: config | |
| 2024-02-26 09:23:33,090 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:23:33,090 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:23:33,090 WARNING SenderThread:196 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) | |
| 2024-02-26 09:23:35,091 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:23:37,198 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:23:40,097 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:23:40,440 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/config.yaml | |
| 2024-02-26 09:23:42,199 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:23:45,217 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:23:47,200 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:23:50,218 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:23:52,201 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:23:55,219 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:23:57,202 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:24:00,220 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:24:02,203 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:24:05,221 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:24:07,204 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:24:10,221 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:24:12,209 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:24:15,222 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:24:17,210 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:24:20,223 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:24:22,211 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:24:25,224 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:24:27,212 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:24:30,225 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:24:32,213 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:24:32,486 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:24:32,488 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:24:35,489 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:24:37,214 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:24:40,490 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:24:42,215 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:24:45,491 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:24:47,219 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:24:50,491 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:24:52,220 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:24:55,492 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:24:57,221 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:25:00,493 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:25:02,226 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:25:02,488 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:25:06,490 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:25:07,227 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:25:07,470 INFO Thread-12 :196 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:25:09,470 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:25:11,847 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:25:12,228 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:25:16,848 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:25:17,229 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:25:21,849 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:25:22,230 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:25:26,850 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:25:27,247 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:25:31,851 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:25:32,284 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:25:32,489 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:25:34,173 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 09:25:34,175 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:25:34,176 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:25:34,176 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:25:34,176 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:25:34,176 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:25:34,176 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:25:34,176 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:25:34,176 DEBUG SenderThread:196 [sender.py:send():382] send: history | |
| 2024-02-26 09:25:34,177 DEBUG SenderThread:196 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 09:25:34,177 INFO SenderThread:196 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 09:25:34,480 INFO Thread-12 :196 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/wandb-summary.json | |
| 2024-02-26 09:25:37,178 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:25:37,295 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:25:42,215 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:25:42,302 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:25:42,484 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/config.yaml | |
| 2024-02-26 09:25:44,419 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 09:25:44,420 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:25:44,421 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:25:44,421 DEBUG SenderThread:196 [sender.py:send():382] send: history | |
| 2024-02-26 09:25:44,421 DEBUG SenderThread:196 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 09:25:44,421 INFO SenderThread:196 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 09:25:44,485 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/wandb-summary.json | |
| 2024-02-26 09:25:47,401 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:25:47,422 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:25:52,424 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:25:52,494 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:25:57,424 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:25:57,510 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:26:02,425 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:26:02,490 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:26:02,510 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:26:07,491 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:26:07,512 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:26:12,498 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:26:12,518 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:26:13,499 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/config.yaml | |
| 2024-02-26 09:26:17,520 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:26:17,670 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:26:22,522 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:26:22,672 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:26:27,523 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:26:27,672 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:26:32,491 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:26:32,524 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:26:33,492 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:26:37,525 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:26:38,492 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:26:42,526 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:26:43,493 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:26:47,527 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:26:48,494 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:26:52,528 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:26:53,495 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:26:57,529 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:26:58,496 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:27:02,492 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:27:02,531 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:27:04,493 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:27:07,532 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:27:09,494 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:27:12,537 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:27:14,495 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:27:17,540 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:27:19,496 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:27:21,524 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:27:22,541 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:27:24,753 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:27:27,542 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:27:29,754 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:27:32,492 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:27:32,543 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:27:35,494 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:27:37,544 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:27:40,495 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:27:42,576 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:27:45,495 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:27:45,907 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 09:27:45,909 DEBUG SenderThread:196 [sender.py:send():382] send: history | |
| 2024-02-26 09:27:45,909 DEBUG SenderThread:196 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 09:27:45,909 INFO SenderThread:196 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 09:27:46,534 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/wandb-summary.json | |
| 2024-02-26 09:27:47,597 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:27:50,911 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:27:52,598 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:27:55,912 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:27:57,599 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:28:00,913 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:28:02,493 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:28:02,603 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:28:06,494 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:28:06,811 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 09:28:06,812 DEBUG SenderThread:196 [sender.py:send():382] send: history | |
| 2024-02-26 09:28:06,812 DEBUG SenderThread:196 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 09:28:06,813 INFO SenderThread:196 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 09:28:07,542 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/wandb-summary.json | |
| 2024-02-26 09:28:07,605 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:28:11,814 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:28:12,698 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:28:16,815 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:28:17,699 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:28:21,816 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:28:22,700 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:28:26,817 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:28:27,701 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:28:31,818 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:28:32,494 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:28:32,703 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:28:37,495 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:28:37,704 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:28:42,496 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:28:42,705 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:28:47,497 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:28:47,706 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:28:52,498 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:28:52,707 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:28:57,499 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:28:57,712 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:29:02,495 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:29:02,713 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:29:03,495 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:29:07,714 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:29:08,496 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:29:12,715 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:29:13,497 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:29:17,716 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:29:18,498 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:29:22,717 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:29:23,499 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:29:27,718 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:29:28,499 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:29:32,495 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:29:32,724 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:29:34,496 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:29:37,725 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:29:39,577 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:29:39,706 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:29:42,726 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:29:44,707 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:29:47,728 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:29:49,708 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:29:52,729 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:29:54,708 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:29:57,750 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:29:59,709 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:30:02,496 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:30:02,778 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:30:02,779 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 09:30:02,780 DEBUG SenderThread:196 [sender.py:send():382] send: history | |
| 2024-02-26 09:30:02,781 DEBUG SenderThread:196 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 09:30:02,781 INFO SenderThread:196 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 09:30:03,587 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/wandb-summary.json | |
| 2024-02-26 09:30:04,782 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:30:07,779 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:30:09,783 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:30:12,781 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:30:14,783 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:30:17,782 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:30:19,784 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:30:22,785 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:30:24,785 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:30:27,785 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:30:29,786 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:30:32,497 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:30:32,787 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:30:33,462 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 09:30:33,463 DEBUG SenderThread:196 [sender.py:send():382] send: history | |
| 2024-02-26 09:30:33,463 DEBUG SenderThread:196 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 09:30:33,464 INFO SenderThread:196 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 09:30:33,596 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/wandb-summary.json | |
| 2024-02-26 09:30:35,464 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:30:38,367 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:30:40,465 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:30:43,368 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:30:45,466 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:30:48,369 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:30:50,467 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:30:53,370 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:30:55,470 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:30:58,371 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:31:00,471 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:31:02,498 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:31:03,372 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:31:05,499 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:31:08,378 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:31:10,500 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:31:13,379 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:31:15,501 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:31:18,385 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:31:20,502 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:31:23,386 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:31:25,503 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:31:28,388 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:31:30,504 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:31:32,498 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:31:33,390 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:31:36,500 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:31:38,391 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:31:41,501 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:31:43,392 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:31:46,501 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:31:48,393 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:31:51,502 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:31:53,394 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:31:55,626 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:31:56,960 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:31:58,395 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:32:01,961 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:32:02,499 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:32:03,396 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:32:07,501 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:32:08,398 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:32:12,501 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:32:13,419 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:32:17,502 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:32:18,456 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:32:19,249 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 09:32:19,251 DEBUG SenderThread:196 [sender.py:send():382] send: history | |
| 2024-02-26 09:32:19,251 DEBUG SenderThread:196 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 09:32:19,252 INFO SenderThread:196 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 09:32:19,636 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/wandb-summary.json | |
| 2024-02-26 09:32:23,253 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:32:23,472 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:32:28,254 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:32:28,475 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:32:32,500 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:32:33,476 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:32:33,501 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:32:38,477 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:32:38,502 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:32:43,478 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:32:43,503 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:32:48,479 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:32:48,504 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:32:53,481 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:32:53,505 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:32:58,482 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:32:58,505 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:33:01,315 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 09:33:01,317 DEBUG SenderThread:196 [sender.py:send():382] send: history | |
| 2024-02-26 09:33:01,317 DEBUG SenderThread:196 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 09:33:01,317 INFO SenderThread:196 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 09:33:01,652 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/wandb-summary.json | |
| 2024-02-26 09:33:02,501 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:33:04,296 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:33:04,502 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:33:09,307 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:33:09,503 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:33:14,308 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:33:14,504 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:33:19,309 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:33:19,505 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:33:24,312 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:33:24,506 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:33:29,313 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:33:29,507 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:33:32,502 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:33:34,314 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:33:35,503 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:33:39,315 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:33:40,504 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:33:44,317 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:33:45,505 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:33:49,318 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:33:50,506 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:33:54,319 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:33:55,507 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:33:59,320 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:34:00,508 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:34:02,503 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:34:04,321 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:34:06,504 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:34:09,322 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:34:11,659 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:34:11,678 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:34:14,323 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:34:16,660 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:34:19,324 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:34:21,661 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:34:24,325 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:34:26,662 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:34:29,336 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:34:31,663 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:34:32,504 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:34:34,368 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:34:36,005 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 09:34:36,007 DEBUG SenderThread:196 [sender.py:send():382] send: history | |
| 2024-02-26 09:34:36,007 DEBUG SenderThread:196 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 09:34:36,007 INFO SenderThread:196 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 09:34:36,009 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-26 09:34:36,011 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:34:36,011 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:34:36,011 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:34:36,012 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:34:36,012 DEBUG SenderThread:196 [sender.py:send():382] send: metric | |
| 2024-02-26 09:34:36,012 DEBUG SenderThread:196 [sender.py:send():382] send: history | |
| 2024-02-26 09:34:36,012 DEBUG SenderThread:196 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-26 09:34:36,013 INFO SenderThread:196 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-26 09:34:36,014 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:34:36,014 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:34:36,014 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:34:36,015 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:34:36,015 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:34:36,015 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:34:36,016 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:34:36,023 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:34:36,023 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:34:36,023 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:34:36,024 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:34:36,688 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/wandb-summary.json | |
| 2024-02-26 09:34:37,024 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:34:39,379 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:34:42,025 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:34:43,927 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:34:43,928 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:34:43,928 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:34:43,928 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:34:43,929 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:34:43,929 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:34:43,930 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:34:43,931 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:34:43,931 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:34:44,384 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:34:47,026 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:34:49,915 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:34:52,027 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:34:54,917 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:34:57,033 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:34:57,695 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/config.yaml | |
| 2024-02-26 09:34:59,918 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:35:02,160 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:35:04,919 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:35:07,160 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:35:09,920 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:35:12,161 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:35:14,921 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:35:17,163 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:35:18,845 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:35:18,845 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:35:18,845 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:35:18,845 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:35:18,846 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:35:18,846 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:35:18,846 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:35:18,846 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:35:18,854 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:35:18,855 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:35:18,855 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:35:18,855 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:35:19,922 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:35:22,856 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:35:24,923 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:35:27,857 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:35:29,924 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:35:32,859 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:35:34,925 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:35:37,860 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:35:39,926 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:35:42,861 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:35:44,927 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:35:47,862 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:35:48,893 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:35:48,893 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:35:48,893 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:35:48,893 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:35:48,894 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:35:48,894 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:35:48,896 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:35:48,898 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:35:48,899 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:35:49,713 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:35:49,933 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:35:50,075 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:35:50,075 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:35:50,075 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:35:50,075 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:35:50,076 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:35:50,076 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:35:50,076 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:35:50,076 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:35:50,083 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:35:50,083 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:35:50,083 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:35:50,084 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:35:51,714 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:35:53,084 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:35:54,940 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:35:58,085 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:35:59,941 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:36:01,568 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:36:01,568 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:36:01,568 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:36:01,568 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:36:01,569 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:36:01,569 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:36:01,569 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:36:01,569 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:36:01,570 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:36:01,570 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:36:01,570 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:36:01,570 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:36:01,571 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:36:01,571 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:36:01,572 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:36:03,573 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:36:04,514 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:36:04,514 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:36:04,515 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:36:04,515 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:36:04,516 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:36:04,516 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:36:04,516 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:36:04,516 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:36:04,516 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:36:04,516 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:36:04,516 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:36:04,517 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:36:04,942 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:36:06,665 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:36:06,665 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:36:06,665 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:36:06,665 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:36:06,666 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:36:06,667 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:36:06,671 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:36:06,672 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:36:06,672 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:36:07,188 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:36:07,188 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:36:07,188 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:36:07,188 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:36:07,188 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:36:07,188 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:36:07,189 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:36:07,189 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:36:07,196 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:36:07,196 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:36:07,196 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:36:07,197 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:36:08,720 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:36:09,197 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:36:09,943 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:36:10,291 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:36:10,291 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:36:10,291 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:36:10,292 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:36:10,293 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:36:10,294 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:36:10,295 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:36:10,296 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:36:10,298 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:36:10,621 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:36:10,621 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:36:10,621 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:36:10,621 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:36:10,621 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:36:10,621 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:36:10,622 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:36:10,623 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:36:10,628 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:36:10,629 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:36:10,629 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:36:10,629 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:36:10,721 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:36:14,630 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:36:14,944 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:36:15,547 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:36:15,547 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:36:15,547 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:36:15,548 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:36:15,549 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:36:15,549 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:36:15,549 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:36:15,549 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:36:15,551 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:36:15,552 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:36:15,552 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:36:15,552 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:36:15,552 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:36:15,553 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:36:15,554 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:36:15,563 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:36:15,563 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:36:17,954 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:36:17,954 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:36:17,955 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:36:17,955 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:36:17,955 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:36:17,956 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:36:17,958 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:36:17,959 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:36:17,959 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:36:19,319 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:36:19,319 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:36:19,319 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:36:19,320 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:36:19,320 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:36:19,321 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:36:19,321 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:36:19,321 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:36:19,328 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:36:19,328 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:36:19,328 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:36:19,329 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:36:19,724 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:36:20,016 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:36:20,330 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:36:21,725 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:36:25,017 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:36:25,331 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:36:30,018 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:36:30,332 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:36:35,019 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:36:35,333 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:36:40,020 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:36:40,334 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:36:42,617 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:36:42,617 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:36:42,617 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:36:42,617 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:36:42,617 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:36:42,620 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:36:42,621 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:36:42,622 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:36:42,623 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:36:45,335 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:36:45,972 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:36:50,336 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:36:50,973 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:36:52,384 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:36:52,384 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:36:52,384 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:36:52,385 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:36:52,385 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:36:52,385 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:36:52,385 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:36:52,385 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:36:52,393 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:36:52,393 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:36:52,393 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:36:52,393 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:36:55,394 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:36:55,974 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:37:00,395 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:37:00,975 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:37:05,396 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:37:05,976 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:37:10,289 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:37:10,289 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:37:10,289 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:37:10,290 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:37:10,290 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:37:10,291 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:37:10,291 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:37:10,292 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:37:10,293 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:37:10,978 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:37:11,284 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:37:11,428 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:37:11,428 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:37:11,428 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:37:11,428 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:37:11,428 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:37:11,428 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:37:11,429 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:37:11,429 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:37:11,436 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:37:11,436 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:37:11,436 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:37:11,437 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:37:11,744 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:37:13,745 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:37:15,979 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:37:16,438 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:37:20,980 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:37:21,439 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:37:24,196 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:37:24,196 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:37:24,197 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:37:24,197 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:37:24,197 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:37:24,198 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:37:24,199 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:37:24,199 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:37:24,199 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:37:24,199 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:37:24,199 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:37:24,200 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:37:24,200 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:37:24,201 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:37:24,203 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:37:24,210 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:37:24,211 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:37:25,982 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:37:27,141 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:37:27,141 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:37:27,141 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:37:27,141 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:37:27,142 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:37:27,142 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:37:27,143 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:37:27,144 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:37:27,144 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:37:27,144 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:37:27,145 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:37:27,145 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:37:27,147 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:37:27,148 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:37:27,148 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:37:27,149 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:37:27,158 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:37:27,158 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:37:27,158 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:37:27,159 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:37:29,521 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:37:29,521 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:37:29,521 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:37:29,521 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:37:29,522 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:37:29,525 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:37:29,527 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:37:29,528 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:37:29,530 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:37:29,842 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:37:29,842 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:37:29,843 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:37:29,843 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:37:29,843 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:37:29,844 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:37:29,844 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:37:29,844 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:37:29,851 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:37:29,851 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:37:29,851 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:37:29,856 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:37:30,983 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:37:31,752 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:37:32,857 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:37:35,709 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:37:35,709 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:37:35,709 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:37:35,709 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:37:35,710 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:37:35,711 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:37:35,712 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:37:35,714 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:37:35,716 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:37:35,984 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:37:36,063 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:37:36,063 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:37:36,063 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:37:36,063 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:37:36,063 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:37:36,064 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:37:36,064 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:37:36,064 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:37:36,071 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:37:36,071 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:37:36,071 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:37:36,072 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:37:37,755 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:37:38,072 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:37:40,256 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:37:40,257 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:37:40,257 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:37:40,257 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:37:40,257 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:37:40,258 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:37:40,259 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:37:40,259 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:37:40,259 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:37:40,259 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:37:40,260 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:37:40,260 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:37:40,260 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:37:40,260 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:37:40,262 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:37:40,262 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:37:40,263 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:37:40,986 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:37:43,264 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:37:43,692 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:37:43,692 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:37:43,692 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:37:43,693 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:37:43,693 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:37:43,694 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:37:43,695 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:37:43,696 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:37:43,697 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:37:44,832 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:37:44,833 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:37:44,833 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:37:44,833 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:37:44,833 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:37:44,834 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:37:44,834 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:37:44,834 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:37:44,841 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:37:44,841 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:37:44,841 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:37:44,841 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:37:45,758 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:37:45,987 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:37:48,842 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:37:50,988 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:37:53,843 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:37:55,989 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:37:58,844 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:38:00,990 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:38:03,845 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:38:05,991 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:38:08,846 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:38:10,992 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:38:13,847 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:38:15,993 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:38:18,848 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:38:20,995 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:38:23,849 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:38:25,996 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:38:28,849 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:38:30,997 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:38:33,850 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:38:35,998 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:38:38,851 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:38:40,999 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:38:43,852 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:38:46,000 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:38:48,853 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:38:51,001 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:38:53,853 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:38:56,002 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:38:58,854 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:39:01,003 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:39:03,855 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:39:06,005 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:39:08,856 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:39:11,006 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:39:13,857 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:39:16,007 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:39:18,858 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:39:21,008 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:39:21,384 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:39:21,385 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:39:21,385 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:39:21,385 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:39:21,385 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:39:21,387 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:39:21,389 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:39:21,390 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:39:21,390 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:39:22,794 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:39:22,794 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:39:22,794 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:39:22,794 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:39:22,795 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:39:22,795 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:39:22,795 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:39:22,797 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:39:22,803 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:39:22,803 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:39:22,803 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:39:22,804 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:39:23,795 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:39:24,805 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:39:25,796 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:39:26,010 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:39:29,805 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:39:31,011 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:39:34,806 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:39:36,012 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:39:39,807 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:39:41,013 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:39:44,807 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:39:46,014 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:39:49,808 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:39:51,015 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:39:54,809 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:39:56,017 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:39:59,809 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:40:01,018 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:40:04,810 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:40:06,019 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:40:09,811 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:40:11,020 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:40:14,812 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:40:16,021 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:40:19,813 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:40:21,022 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:40:24,814 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:40:26,023 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:40:29,814 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:40:31,025 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:40:34,815 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:40:36,026 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:40:39,816 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:40:41,027 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:40:44,817 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:40:46,028 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:40:49,818 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:40:51,029 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:40:54,819 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:40:56,030 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:40:59,820 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:41:01,032 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:41:04,820 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:41:06,032 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:41:09,821 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:41:11,034 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:41:13,198 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:41:13,199 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:41:13,199 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:41:13,199 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:41:13,200 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:41:13,200 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:41:13,201 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:41:13,202 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:41:13,203 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:41:14,706 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-26 09:41:14,706 INFO HandlerThread:196 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-26 09:41:14,707 INFO HandlerThread:196 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-26 09:41:14,707 DEBUG SystemMonitor:196 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-26 09:41:14,707 DEBUG SystemMonitor:196 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-26 09:41:14,711 DEBUG SystemMonitor:196 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-26 09:41:14,707 INFO HandlerThread:196 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-26 09:41:14,712 INFO HandlerThread:196 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-26 09:41:14,716 INFO HandlerThread:196 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-26 09:41:14,717 INFO HandlerThread:196 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-26 09:41:14,717 INFO HandlerThread:196 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-26 09:41:14,717 DEBUG SenderThread:196 [sender.py:send():382] send: stats | |
| 2024-02-26 09:41:15,718 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:41:16,059 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:41:16,838 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:41:17,838 INFO Thread-12 :196 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240226_092105-91uhs91a/files/output.log | |
| 2024-02-26 09:41:20,719 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:41:21,061 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:41:25,720 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:41:26,062 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:41:30,721 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:41:31,063 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:41:35,722 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:41:36,064 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:41:40,723 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:41:41,065 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:41:44,004 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-26 09:41:44,004 INFO HandlerThread:196 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-26 09:41:44,004 INFO HandlerThread:196 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-26 09:41:44,005 INFO SystemMonitor:196 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-26 09:41:44,006 INFO SystemMonitor:196 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-26 09:41:44,008 INFO SystemMonitor:196 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-26 09:41:44,010 INFO SystemMonitor:196 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-26 09:41:44,011 INFO SystemMonitor:196 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-26 09:41:44,012 INFO SystemMonitor:196 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-26 09:41:45,724 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:41:46,155 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |
| 2024-02-26 09:41:50,725 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-26 09:41:51,279 DEBUG HandlerThread:196 [handler.py:handle_request():146] handle_request: keepalive | |