| 2024-02-14 12:54:29,380 INFO StreamThr :35088 [internal.py:wandb_internal():86] W&B internal server running at pid: 35088, started at: 2024-02-14 12:54:29.379470 | |
| 2024-02-14 12:54:29,381 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status | |
| 2024-02-14 12:54:30,447 INFO WriterThread:35088 [datastore.py:open_for_write():87] open: /content/wandb/run-20240214_125429-fe3s7gan/run-fe3s7gan.wandb | |
| 2024-02-14 12:54:30,447 DEBUG SenderThread:35088 [sender.py:send():382] send: header | |
| 2024-02-14 12:54:30,450 DEBUG SenderThread:35088 [sender.py:send():382] send: run | |
| 2024-02-14 12:54:30,674 INFO SenderThread:35088 [dir_watcher.py:__init__():211] watching files in: /content/wandb/run-20240214_125429-fe3s7gan/files | |
| 2024-02-14 12:54:30,674 INFO SenderThread:35088 [sender.py:_start_run_threads():1136] run started: fe3s7gan with start time 1707915269.380721 | |
| 2024-02-14 12:54:30,681 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-02-14 12:54:30,682 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: check_version | |
| 2024-02-14 12:54:30,815 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-02-14 12:54:30,824 DEBUG HandlerThread:35088 [system_info.py:__init__():27] System info init | |
| 2024-02-14 12:54:30,824 DEBUG HandlerThread:35088 [system_info.py:__init__():42] System info init done | |
| 2024-02-14 12:54:30,824 INFO HandlerThread:35088 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-14 12:54:30,824 INFO SystemMonitor:35088 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-14 12:54:30,825 INFO HandlerThread:35088 [system_monitor.py:probe():214] Collecting system info | |
| 2024-02-14 12:54:30,825 INFO SystemMonitor:35088 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-14 12:54:30,826 INFO SystemMonitor:35088 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-14 12:54:30,826 INFO SystemMonitor:35088 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-14 12:54:30,828 INFO SystemMonitor:35088 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-14 12:54:30,828 INFO SystemMonitor:35088 [interfaces.py:start():190] Started network monitoring | |
| 2024-02-14 12:54:30,832 DEBUG HandlerThread:35088 [system_info.py:probe():151] Probing system | |
| 2024-02-14 12:54:30,833 DEBUG HandlerThread:35088 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-02-14 12:54:30,833 DEBUG HandlerThread:35088 [system_info.py:probe():199] Probing system done | |
| 2024-02-14 12:54:30,833 DEBUG HandlerThread:35088 [system_monitor.py:probe():223] {'os': 'Linux-6.1.58+-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2024-02-14T12:54:30.832517', 'startedAt': '2024-02-14T12:54:29.373808', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'Smart_Agriculture_Falcon.ipynb', 'codePathLocal': None, 'colab': 'https://colab.research.google.com/notebook#fileId=1kv2VO7q0c8xX_DQt5NERLeJJZtHW6zHO', 'host': 'f4b889916ade', 'username': 'root', 'executable': '/usr/bin/python3', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 2299.998, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}, {'current': 2299.998, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 166.77415084838867, 'used': 66.21718978881836}}, 'gpu': 'Tesla T4', 'gpu_count': 1, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 50.993690490722656}} | |
| 2024-02-14 12:54:30,834 INFO HandlerThread:35088 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-02-14 12:54:30,834 INFO HandlerThread:35088 [system_monitor.py:probe():227] Publishing system info | |
| 2024-02-14 12:54:30,834 INFO HandlerThread:35088 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-02-14 12:54:30,840 DEBUG SenderThread:35088 [sender.py:send():382] send: files | |
| 2024-02-14 12:54:30,840 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now | |
| 2024-02-14 12:54:30,856 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-02-14 12:54:30,856 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: python_packages | |
| 2024-02-14 12:54:30,896 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:54:30,896 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:54:30,897 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:54:31,103 DEBUG SenderThread:35088 [sender.py:send():382] send: telemetry | |
| 2024-02-14 12:54:31,103 DEBUG SenderThread:35088 [sender.py:send():382] send: config | |
| 2024-02-14 12:54:31,104 DEBUG SenderThread:35088 [sender.py:send():382] send: metric | |
| 2024-02-14 12:54:31,104 DEBUG SenderThread:35088 [sender.py:send():382] send: telemetry | |
| 2024-02-14 12:54:31,104 DEBUG SenderThread:35088 [sender.py:send():382] send: metric | |
| 2024-02-14 12:54:31,104 WARNING SenderThread:35088 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) | |
| 2024-02-14 12:54:31,104 DEBUG SenderThread:35088 [sender.py:send():382] send: telemetry | |
| 2024-02-14 12:54:31,178 INFO wandb-upload_0:35088 [upload_job.py:push():131] Uploaded file /tmp/tmp668dgil2wandb/pe1jrqzh-wandb-metadata.json | |
| 2024-02-14 12:54:31,675 INFO Thread-12 :35088 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240214_125429-fe3s7gan/files/requirements.txt | |
| 2024-02-14 12:54:31,675 INFO Thread-12 :35088 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-metadata.json | |
| 2024-02-14 12:54:35,105 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:54:40,106 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:54:45,107 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:54:45,857 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:54:45,857 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:54:45,897 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:54:50,987 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:54:55,988 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:00,857 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:55:00,857 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:55:00,897 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:55:01,012 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:01,684 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/config.yaml | |
| 2024-02-14 12:55:06,125 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:11,126 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:15,857 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:55:15,857 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:55:15,897 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:55:16,924 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:21,925 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:26,926 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:30,828 DEBUG SystemMonitor:35088 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-02-14 12:55:30,829 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 12:55:30,857 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:55:30,857 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:55:30,897 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:55:31,981 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:36,982 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:41,983 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:45,857 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:55:45,858 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:55:45,898 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:55:47,973 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:51,040 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 12:55:51,042 DEBUG SenderThread:35088 [sender.py:send():382] send: metric | |
| 2024-02-14 12:55:51,042 DEBUG SenderThread:35088 [sender.py:send():382] send: metric | |
| 2024-02-14 12:55:51,042 DEBUG SenderThread:35088 [sender.py:send():382] send: metric | |
| 2024-02-14 12:55:51,042 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 12:55:51,043 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 12:55:51,045 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 12:55:51,700 INFO Thread-12 :35088 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 12:55:51,701 INFO Thread-12 :35088 [dir_watcher.py:_on_file_created():271] file/dir created: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 12:55:53,082 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:55:53,701 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 12:55:58,083 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:56:00,830 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 12:56:00,857 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:56:00,858 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:56:00,858 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:56:03,930 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:56:04,705 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/config.yaml | |
| 2024-02-14 12:56:09,031 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:56:14,032 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:56:15,857 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:56:15,858 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:56:15,898 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:56:19,911 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:56:24,912 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:56:29,913 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:56:30,831 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 12:56:30,857 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:56:30,858 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:56:30,898 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:56:34,993 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:56:39,995 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:56:44,996 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:56:45,857 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:56:45,858 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:56:45,898 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:56:50,952 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:56:55,953 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:57:00,832 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 12:57:00,858 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:57:00,858 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:57:00,898 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:57:01,923 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:57:06,924 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:57:11,925 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:57:14,440 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 12:57:14,441 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 12:57:14,442 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 12:57:14,443 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 12:57:14,729 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 12:57:15,729 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 12:57:15,882 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:57:15,882 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:57:15,919 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:57:17,015 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:57:22,016 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:57:27,017 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:57:30,832 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 12:57:30,864 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:57:30,865 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:57:30,906 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:57:32,019 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:57:37,020 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:57:42,021 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:57:45,865 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:57:45,865 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:57:45,905 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:57:47,950 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:57:52,951 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:57:57,952 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:58:00,833 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 12:58:00,865 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:58:00,866 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:58:00,906 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:58:03,014 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:58:08,016 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:58:13,017 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:58:15,865 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:58:15,866 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:58:15,906 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:58:18,936 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:58:23,936 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:58:28,937 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:58:30,834 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 12:58:30,865 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:58:30,866 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:58:30,906 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:58:34,535 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:58:36,001 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 12:58:36,002 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 12:58:36,002 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 12:58:36,003 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 12:58:36,756 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 12:58:37,757 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 12:58:40,043 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:58:45,044 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:58:45,866 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:58:45,866 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:58:45,889 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:58:50,945 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:58:55,946 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:59:00,834 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 12:59:00,866 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:59:00,866 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:59:00,889 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:59:01,002 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:59:06,003 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:59:11,004 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:59:15,866 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:59:15,866 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:59:15,890 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:59:16,956 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:59:21,957 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:59:26,958 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:59:30,835 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 12:59:30,866 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:59:30,866 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:59:30,890 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:59:32,933 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:59:37,934 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:59:42,934 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:59:45,866 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 12:59:45,867 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 12:59:45,907 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 12:59:48,002 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:59:53,003 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:59:58,004 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 12:59:59,754 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 12:59:59,755 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 12:59:59,755 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 12:59:59,756 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 12:59:59,782 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:00:00,836 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:00:00,888 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:00:00,888 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:00:00,910 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:00:01,783 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 13:00:03,973 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:00:08,974 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:00:13,975 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:00:15,870 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:00:15,870 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:00:15,912 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:00:19,960 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:00:24,961 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:00:29,962 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:00:30,837 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:00:30,870 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:00:30,871 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:00:30,911 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:00:35,005 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:00:40,006 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:00:45,007 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:00:45,870 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:00:45,871 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:00:45,894 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:00:50,015 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:00:55,016 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:01:00,017 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:01:00,838 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:01:00,870 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:01:00,871 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:01:00,895 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:01:05,029 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:01:10,030 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:01:15,031 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:01:15,870 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:01:15,871 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:01:15,894 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:01:20,962 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:01:23,085 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:01:23,086 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:01:23,086 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:01:23,087 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:01:23,810 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:01:25,810 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 13:01:26,126 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:01:30,838 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:01:30,871 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:01:30,871 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:01:30,894 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:01:31,984 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:01:36,985 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:01:41,986 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:01:45,871 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:01:45,871 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:01:45,895 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:01:47,963 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:01:52,964 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:01:57,964 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:02:00,839 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:02:00,871 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:02:00,871 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:02:00,912 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:02:02,983 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:02:07,984 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:02:12,985 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:02:15,871 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:02:15,872 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:02:15,895 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:02:18,964 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:02:23,965 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:02:28,966 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:02:30,840 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:02:30,871 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:02:30,872 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:02:30,912 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:02:34,032 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:02:39,033 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:02:44,034 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:02:45,871 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:02:45,872 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:02:45,895 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:02:47,687 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:02:47,688 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:02:47,688 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:02:47,689 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:02:47,837 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:02:49,690 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:02:49,838 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 13:02:54,691 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:02:59,692 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:03:00,840 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:03:00,872 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:03:00,872 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:03:00,895 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:03:05,036 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:03:10,037 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:03:15,038 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:03:15,872 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:03:15,872 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:03:15,895 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:03:21,022 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:03:26,023 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:03:30,841 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:03:30,872 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:03:30,872 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:03:30,895 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:03:31,028 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:03:36,029 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:03:41,030 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:03:45,872 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:03:45,873 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:03:45,896 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:03:46,966 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:03:51,967 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:03:56,968 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:04:00,842 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:04:00,872 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:04:00,873 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:04:00,913 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:04:01,989 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:04:06,990 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:04:11,991 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:04:12,712 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:04:12,713 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:04:12,713 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:04:12,714 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:04:12,866 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:04:13,866 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 13:04:15,872 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:04:15,873 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:04:15,896 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:04:17,976 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:04:22,977 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:04:27,978 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:04:30,843 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:04:30,872 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:04:30,873 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:04:30,896 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:04:33,029 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:04:38,030 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:04:43,031 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:04:45,873 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:04:45,873 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:04:45,896 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:04:48,950 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:04:53,951 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:04:58,951 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:05:00,844 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:05:00,873 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:05:00,873 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:05:00,913 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:05:04,947 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:05:09,947 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:05:14,948 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:05:15,873 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:05:15,874 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:05:15,914 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:05:19,973 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:05:24,974 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:05:29,974 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:05:30,844 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:05:30,873 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:05:30,874 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:05:30,896 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:05:33,919 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:05:33,920 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:05:33,920 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:05:33,921 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:05:34,896 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:05:35,896 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 13:05:35,922 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:05:40,922 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:05:45,873 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:05:45,874 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:05:45,897 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:05:46,022 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:05:51,022 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:05:56,023 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:06:00,845 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:06:00,873 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:06:00,874 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:06:00,897 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:06:01,949 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:06:06,950 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:06:11,951 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:06:15,873 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:06:15,874 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:06:15,914 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:06:16,988 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:06:21,989 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:06:26,990 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:06:30,846 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:06:30,874 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:06:30,874 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:06:30,897 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:06:32,030 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:06:37,031 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:06:42,031 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:06:45,874 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:06:45,874 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:06:45,897 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:06:48,010 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:06:53,010 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:06:56,249 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:06:56,250 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:06:56,251 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:06:56,251 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:06:56,924 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:06:57,924 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 13:06:58,252 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:07:00,847 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:07:00,874 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:07:00,874 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:07:00,897 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:07:03,957 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:07:08,958 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:07:13,959 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:07:15,874 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:07:15,874 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:07:15,915 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:07:19,010 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:07:24,011 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:07:29,012 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:07:30,847 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:07:30,874 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:07:30,874 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:07:30,897 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:07:34,025 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:07:39,026 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:07:44,027 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:07:45,874 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:07:45,875 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:07:45,898 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:07:49,027 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:07:54,028 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:07:59,029 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:08:00,848 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:08:00,874 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:08:00,875 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:08:00,898 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:08:04,981 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:08:09,982 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:08:14,983 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:08:15,874 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:08:15,875 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:08:15,898 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:08:18,059 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:08:18,060 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:08:18,060 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:08:18,061 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:08:18,951 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:08:19,951 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 13:08:20,101 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:08:25,102 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:08:30,103 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:08:30,849 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:08:30,875 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:08:30,875 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:08:30,898 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:08:36,027 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:08:41,028 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:08:45,875 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:08:45,875 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:08:45,898 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:08:46,954 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:08:51,955 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:08:56,956 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:09:00,855 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:09:00,875 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:09:00,875 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:09:00,915 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:09:01,985 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:09:06,986 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:09:11,987 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:09:15,875 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:09:15,875 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:09:15,899 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:09:16,992 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:09:21,993 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:09:26,994 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:09:30,850 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:09:30,875 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:09:30,876 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:09:30,899 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:09:32,961 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:09:37,962 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:09:42,963 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:09:43,076 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:09:43,077 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:09:43,077 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:09:43,079 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:09:43,978 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:09:44,978 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 13:09:45,875 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:09:45,876 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:09:45,899 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:09:48,010 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:09:53,011 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:09:58,011 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:10:00,851 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:10:00,875 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:10:00,876 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:10:00,899 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:10:03,018 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:10:08,019 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:10:13,020 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:10:15,876 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:10:15,876 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:10:15,899 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:10:18,959 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:10:23,960 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:10:28,961 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:10:30,853 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:10:30,876 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:10:30,876 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:10:30,916 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:10:33,995 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:10:38,996 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:10:43,997 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:10:45,876 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:10:45,876 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:10:45,899 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:10:49,952 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:10:54,953 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:10:59,954 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:11:00,852 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:11:00,876 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:11:00,876 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:11:00,917 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:11:04,844 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:11:04,845 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:11:04,846 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:11:04,846 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:11:05,005 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:11:05,847 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:11:07,006 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 13:11:10,848 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:11:15,848 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:11:15,876 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:11:15,876 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:11:15,900 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:11:20,954 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:11:25,955 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:11:30,853 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:11:30,876 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:11:30,877 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:11:30,917 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:11:31,946 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:11:36,948 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:11:41,949 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:11:45,876 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:11:45,877 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:11:45,917 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:11:46,955 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:11:51,955 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:11:56,956 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:12:00,854 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:12:00,876 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:12:00,877 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:12:00,917 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:12:01,998 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:12:06,999 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:12:12,000 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:12:15,877 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:12:15,877 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:12:15,900 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:12:17,988 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:12:22,989 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:12:26,111 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:12:26,113 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:12:26,113 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:12:26,114 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:12:27,033 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:12:27,033 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 13:12:28,115 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:12:30,854 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:12:30,877 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:12:30,877 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:12:30,900 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:12:33,968 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:12:38,969 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:12:43,969 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:12:45,877 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:12:45,877 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:12:45,917 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:12:49,968 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:12:54,969 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:12:59,970 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:13:00,855 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:13:00,877 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:13:00,877 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:13:00,918 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:13:05,011 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:13:10,012 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:13:15,013 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:13:15,877 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:13:15,878 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:13:15,901 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:13:20,975 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:13:25,976 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:13:30,856 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:13:30,877 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:13:30,878 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:13:30,901 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:13:31,953 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:13:36,954 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:13:41,955 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:13:45,877 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:13:45,878 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:13:45,918 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:13:47,935 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:13:49,704 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:13:49,705 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:13:49,706 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:13:49,706 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:13:50,061 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:13:51,061 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 13:13:53,746 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:13:58,747 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:14:00,857 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:14:00,878 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:14:00,878 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:14:00,901 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:14:03,979 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:14:08,980 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:14:13,981 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:14:15,878 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:14:15,878 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:14:15,901 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:14:19,007 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:14:24,008 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:14:29,009 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:14:30,858 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:14:30,878 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:14:30,878 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:14:30,902 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:14:34,027 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:14:39,028 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:14:44,029 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:14:45,878 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:14:45,878 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:14:45,902 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:14:49,974 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:14:54,975 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:14:59,976 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:15:00,859 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:15:00,878 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:15:00,878 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:15:00,919 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:15:05,939 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:15:10,940 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:15:11,165 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:15:11,166 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:15:11,166 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:15:11,167 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:15:12,088 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:15:13,088 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 13:15:15,878 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:15:15,879 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:15:15,902 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:15:15,945 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:15:20,946 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:15:25,947 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:15:30,859 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:15:30,878 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:15:30,879 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:15:30,919 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:15:31,029 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:15:36,030 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:15:41,031 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:15:45,879 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:15:45,879 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:15:45,902 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:15:46,040 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:15:51,041 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:15:56,041 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:16:00,860 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:16:00,879 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:16:00,879 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:16:00,902 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:16:02,031 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:16:07,032 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:16:12,033 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:16:15,879 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:16:15,879 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:16:15,902 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:16:17,990 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:16:22,991 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:16:27,992 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:16:30,861 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:16:30,879 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:16:30,880 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:16:30,902 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:16:33,950 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:16:35,337 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:16:35,338 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:16:35,339 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:16:35,339 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:16:36,116 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:16:37,117 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 13:16:39,379 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:16:44,380 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:16:45,879 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:16:45,880 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:16:45,903 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:16:50,026 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:16:55,027 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:17:00,028 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:17:00,861 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:17:00,879 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:17:00,880 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:17:00,903 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:17:05,979 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:17:10,980 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:17:15,880 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:17:15,880 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:17:15,920 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:17:16,940 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:17:21,941 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:17:26,942 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:17:30,862 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:17:30,880 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:17:30,880 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:17:30,920 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:17:32,017 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:17:37,018 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:17:42,019 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:17:45,880 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:17:45,880 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:17:45,903 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:17:47,973 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:17:52,974 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:17:57,919 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:17:57,920 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:17:57,920 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:17:57,920 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:17:58,142 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:17:58,959 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:17:59,143 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 13:18:00,863 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:18:00,880 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:18:00,880 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:18:00,903 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:18:03,984 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:18:08,985 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:18:13,986 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:18:15,880 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:18:15,880 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:18:15,903 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:18:19,958 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:18:24,959 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:18:29,960 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:18:30,864 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:18:30,880 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:18:30,881 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:18:30,921 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:18:35,029 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:18:40,030 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:18:45,031 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:18:45,880 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:18:45,881 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:18:45,904 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:18:50,958 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:18:55,959 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:19:00,864 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:19:00,880 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:19:00,881 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:19:00,921 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:19:00,982 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:19:05,983 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:19:10,984 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:19:15,881 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:19:15,881 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:19:15,904 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:19:16,965 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:19:20,226 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:19:20,227 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:19:20,228 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:19:20,228 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:19:21,168 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:19:21,168 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/output.log | |
| 2024-02-14 13:19:22,268 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:19:27,269 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:19:30,865 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:19:30,881 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:19:30,881 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:19:30,904 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:19:32,949 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:19:37,950 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:19:42,951 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:19:45,881 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:19:45,881 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:19:45,922 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:19:48,012 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:19:53,013 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:19:58,014 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:20:00,866 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:20:00,881 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:20:00,882 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:20:00,904 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:20:03,053 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:20:08,053 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:20:13,054 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:20:15,881 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:20:15,882 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:20:15,904 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:20:18,979 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:20:23,980 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:20:28,980 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:20:30,867 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:20:30,882 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:20:30,882 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:20:30,922 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:20:34,968 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:20:39,969 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:20:43,027 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:20:43,028 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:20:43,029 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:20:43,029 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:20:43,195 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:20:45,030 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:20:45,882 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:20:45,882 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:20:45,905 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:20:50,988 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:20:55,989 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:21:00,867 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:21:00,882 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:21:00,882 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:21:00,905 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:21:01,023 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:21:06,023 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:21:11,025 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:21:15,882 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:21:15,882 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:21:15,905 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:21:16,060 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:21:21,061 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:21:26,062 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:21:30,868 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:21:30,882 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:21:30,882 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:21:30,905 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:21:32,051 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:21:37,052 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:21:42,053 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:21:45,882 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:21:45,883 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:21:45,905 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:21:47,960 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:21:52,961 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:21:57,962 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:22:00,869 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:22:00,882 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:22:00,883 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:22:00,923 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:22:02,963 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:22:06,595 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:22:06,597 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:22:06,597 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:22:06,597 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:22:07,223 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:22:08,598 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:22:13,599 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:22:15,883 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:22:15,883 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:22:15,905 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:22:18,987 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:22:23,988 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:22:28,989 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:22:30,869 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:22:30,883 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:22:30,883 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:22:30,923 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:22:34,015 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:22:39,016 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:22:44,017 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:22:45,883 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:22:45,883 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:22:45,906 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:22:49,994 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:22:54,995 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:22:59,996 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:23:00,870 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:23:00,883 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:23:00,883 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:23:00,906 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:23:05,966 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:23:10,967 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:23:15,883 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:23:15,884 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:23:15,924 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:23:16,967 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:23:21,968 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:23:26,969 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:23:30,871 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:23:30,883 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:23:30,884 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:23:30,924 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:23:32,040 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:23:33,594 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:23:33,596 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:23:33,596 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:23:33,596 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:23:34,251 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:23:37,597 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:23:42,598 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:23:45,883 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:23:45,884 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:23:45,906 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:23:48,051 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:23:53,051 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:23:58,052 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:24:00,872 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:24:00,884 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:24:00,884 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:24:00,906 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:24:03,950 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:24:08,951 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:24:13,952 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:24:15,884 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:24:15,884 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:24:15,924 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:24:19,016 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:24:24,017 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:24:29,018 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:24:30,873 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:24:30,884 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:24:30,884 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:24:30,906 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:24:34,979 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:24:39,980 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:24:44,981 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:24:45,884 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:24:45,885 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:24:45,925 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:24:50,038 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:24:55,039 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:24:58,861 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:24:58,862 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:24:58,862 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:24:58,863 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:24:59,278 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:25:00,863 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:25:00,873 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:25:00,884 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:25:00,885 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:25:00,907 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:25:06,032 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:25:11,033 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:25:15,884 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:25:15,885 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:25:15,907 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:25:17,013 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:25:22,015 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:25:27,015 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:25:30,874 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:25:30,884 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:25:30,885 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:25:30,925 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:25:32,041 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:25:37,042 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:25:42,043 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:25:45,885 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:25:45,885 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:25:45,907 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:25:47,981 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:25:52,982 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:25:57,983 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:26:00,875 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:26:00,885 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:26:00,885 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:26:00,925 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:26:03,981 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:26:08,982 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:26:13,983 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:26:15,885 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:26:15,885 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:26:15,926 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:26:19,982 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:26:22,290 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:26:22,292 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:26:22,292 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:26:22,292 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:26:22,306 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:26:25,293 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:26:30,294 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:26:30,875 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:26:30,885 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:26:30,885 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:26:30,908 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:26:35,994 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:26:40,995 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:26:45,885 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:26:45,886 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:26:45,926 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:26:46,013 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:26:51,014 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:26:56,015 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:27:00,881 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:27:00,885 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:27:00,886 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:27:00,926 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:27:02,002 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:27:07,003 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:27:12,004 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:27:15,885 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:27:15,886 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:27:15,926 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:27:17,011 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:27:22,012 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:27:27,013 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:27:30,877 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:27:30,885 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:27:30,886 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:27:30,926 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:27:32,039 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:27:37,040 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:27:42,041 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:27:45,886 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:27:45,886 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:27:45,908 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:27:46,323 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:27:46,325 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:27:46,325 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:27:46,327 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:27:46,334 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:27:47,327 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:27:52,328 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:27:57,329 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:28:00,878 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:28:00,886 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:28:00,886 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:28:00,908 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:28:03,008 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:28:08,009 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:28:13,010 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:28:15,886 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:28:15,886 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:28:15,927 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:28:18,987 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:28:23,989 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:28:28,990 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:28:30,878 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:28:30,886 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:28:30,886 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:28:30,927 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:28:34,989 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:28:39,990 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:28:44,992 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:28:45,886 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:28:45,887 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:28:45,927 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:28:50,986 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:28:55,987 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:29:00,879 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:29:00,886 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:29:00,887 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:29:00,927 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:29:04,761 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:29:09,761 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:29:10,743 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:29:10,744 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:29:10,744 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:29:10,745 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:29:11,360 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:29:15,030 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-02-14 13:29:15,031 DEBUG SenderThread:35088 [sender.py:send():382] send: metric | |
| 2024-02-14 13:29:15,032 DEBUG SenderThread:35088 [sender.py:send():382] send: metric | |
| 2024-02-14 13:29:15,032 DEBUG SenderThread:35088 [sender.py:send():382] send: metric | |
| 2024-02-14 13:29:15,032 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:29:15,032 DEBUG SenderThread:35088 [sender.py:send():382] send: metric | |
| 2024-02-14 13:29:15,033 DEBUG SenderThread:35088 [sender.py:send():382] send: metric | |
| 2024-02-14 13:29:15,033 DEBUG SenderThread:35088 [sender.py:send():382] send: history | |
| 2024-02-14 13:29:15,033 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: summary_record | |
| 2024-02-14 13:29:15,034 INFO SenderThread:35088 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end | |
| 2024-02-14 13:29:15,361 INFO Thread-12 :35088 [dir_watcher.py:_on_file_modified():288] file/dir modified: /content/wandb/run-20240214_125429-fe3s7gan/files/wandb-summary.json | |
| 2024-02-14 13:29:15,886 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-02-14 13:29:15,887 DEBUG SenderThread:35088 [sender.py:send_request():409] send_request: stop_status | |
| 2024-02-14 13:29:15,909 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-02-14 13:29:20,440 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: pause | |
| 2024-02-14 13:29:20,441 INFO HandlerThread:35088 [handler.py:handle_request_pause():708] stopping system metrics thread | |
| 2024-02-14 13:29:20,441 INFO HandlerThread:35088 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-02-14 13:29:20,441 INFO HandlerThread:35088 [interfaces.py:finish():202] Joined cpu monitor | |
| 2024-02-14 13:29:20,441 DEBUG SystemMonitor:35088 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-02-14 13:29:20,442 DEBUG SystemMonitor:35088 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-02-14 13:29:20,442 INFO HandlerThread:35088 [interfaces.py:finish():202] Joined disk monitor | |
| 2024-02-14 13:29:20,450 INFO HandlerThread:35088 [interfaces.py:finish():202] Joined gpu monitor | |
| 2024-02-14 13:29:20,450 INFO HandlerThread:35088 [interfaces.py:finish():202] Joined memory monitor | |
| 2024-02-14 13:29:20,450 INFO HandlerThread:35088 [interfaces.py:finish():202] Joined network monitor | |
| 2024-02-14 13:29:20,450 DEBUG SenderThread:35088 [sender.py:send():382] send: stats | |
| 2024-02-14 13:29:20,451 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-02-14 13:29:20,481 DEBUG HandlerThread:35088 [handler.py:handle_request():146] handle_request: resume | |
| 2024-02-14 13:29:20,481 INFO HandlerThread:35088 [handler.py:handle_request_resume():699] starting system metrics thread | |
| 2024-02-14 13:29:20,481 INFO HandlerThread:35088 [system_monitor.py:start():194] Starting system monitor | |
| 2024-02-14 13:29:20,481 INFO SystemMonitor:35088 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-02-14 13:29:20,481 INFO SystemMonitor:35088 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-02-14 13:29:20,482 INFO SystemMonitor:35088 [interfaces.py:start():190] Started disk monitoring | |
| 2024-02-14 13:29:20,482 INFO SystemMonitor:35088 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-02-14 13:29:20,483 INFO SystemMonitor:35088 [interfaces.py:start():190] Started memory monitoring | |
| 2024-02-14 13:29:20,483 INFO SystemMonitor:35088 [interfaces.py:start():190] Started network monitoring | |