| 2024-09-17 12:27:26,830 INFO StreamThr :4132313 [internal.py:wandb_internal():85] W&B internal server running at pid: 4132313, started at: 2024-09-17 12:27:26.826981 | |
| 2024-09-17 12:27:26,834 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status | |
| 2024-09-17 12:27:26,838 INFO WriterThread:4132313 [datastore.py:open_for_write():87] open: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/run-z0ntw6ob.wandb | |
| 2024-09-17 12:27:26,844 DEBUG SenderThread:4132313 [sender.py:send():391] send: header | |
| 2024-09-17 12:27:26,848 DEBUG SenderThread:4132313 [sender.py:send():391] send: run | |
| 2024-09-17 12:27:27,198 INFO SenderThread:4132313 [dir_watcher.py:__init__():211] watching files in: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files | |
| 2024-09-17 12:27:27,198 INFO SenderThread:4132313 [sender.py:_start_run_threads():1200] run started: z0ntw6ob with start time 1726572446.82874 | |
| 2024-09-17 12:27:27,214 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: run_start | |
| 2024-09-17 12:27:27,243 DEBUG HandlerThread:4132313 [system_info.py:__init__():26] System info init | |
| 2024-09-17 12:27:27,243 DEBUG HandlerThread:4132313 [system_info.py:__init__():41] System info init done | |
| 2024-09-17 12:27:27,244 INFO HandlerThread:4132313 [system_monitor.py:start():194] Starting system monitor | |
| 2024-09-17 12:27:27,244 INFO SystemMonitor:4132313 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-09-17 12:27:27,244 INFO HandlerThread:4132313 [system_monitor.py:probe():214] Collecting system info | |
| 2024-09-17 12:27:27,247 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started cpu monitoring | |
| 2024-09-17 12:27:27,250 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started disk monitoring | |
| 2024-09-17 12:27:27,252 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started gpu monitoring | |
| 2024-09-17 12:27:27,253 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started memory monitoring | |
| 2024-09-17 12:27:27,257 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started network monitoring | |
| 2024-09-17 12:27:27,284 DEBUG HandlerThread:4132313 [system_info.py:probe():152] Probing system | |
| 2024-09-17 12:27:27,290 DEBUG HandlerThread:4132313 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2024-09-17 12:27:27,290 DEBUG HandlerThread:4132313 [system_info.py:probe():200] Probing system done | |
| 2024-09-17 12:27:27,290 DEBUG HandlerThread:4132313 [system_monitor.py:probe():223] {'os': 'Linux-4.18.0-513.24.1.el8_9.x86_64-x86_64-with-glibc2.28', 'python': '3.11.4', 'heartbeatAt': '2024-09-17T11:27:27.284628', 'startedAt': '2024-09-17T11:27:26.786345', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': '<python with no main file>', 'codePathLocal': None, 'host': 'codon-gpu-006.ebi.ac.uk', 'username': 'amrufai', 'executable': '/nfs/production/literature/amina-mardiyyah/envs/llm-prompt/bin/python', 'cpu_count': 48, 'cpu_count_logical': 48, 'cpu_freq': {'current': 2758.33775, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 1157.924, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}, {'current': 2800.0, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 47.760292053222656, 'used': 15.848060607910156}}, 'gpu': 'NVIDIA A100 80GB PCIe', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA A100 80GB PCIe', 'memory_total': 85899345920}], 'memory': {'total': 502.8375930786133}} | |
| 2024-09-17 12:27:27,291 INFO HandlerThread:4132313 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-09-17 12:27:27,291 INFO HandlerThread:4132313 [system_monitor.py:probe():227] Publishing system info | |
| 2024-09-17 12:27:27,296 INFO HandlerThread:4132313 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-09-17 12:27:27,320 DEBUG SenderThread:4132313 [sender.py:send():391] send: files | |
| 2024-09-17 12:27:27,320 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-metadata.json with policy now | |
| 2024-09-17 12:27:27,845 INFO wandb-upload_0:4132313 [upload_job.py:push():130] Uploaded file /tmp/tmpr_ev51fvwandb/7xtqy0my-wandb-metadata.json | |
| 2024-09-17 12:27:27,876 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: python_packages | |
| 2024-09-17 12:27:27,877 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:27:27,877 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: python_packages | |
| 2024-09-17 12:27:27,880 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:27:27,883 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:27:27,923 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: pause | |
| 2024-09-17 12:27:27,923 INFO HandlerThread:4132313 [handler.py:handle_request_pause():728] stopping system metrics thread | |
| 2024-09-17 12:27:27,923 INFO HandlerThread:4132313 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-09-17 12:27:27,924 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-09-17 12:27:27,925 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined cpu monitor | |
| 2024-09-17 12:27:27,926 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-09-17 12:27:27,927 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-09-17 12:27:27,927 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined disk monitor | |
| 2024-09-17 12:27:28,000 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined gpu monitor | |
| 2024-09-17 12:27:28,000 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined memory monitor | |
| 2024-09-17 12:27:28,001 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined network monitor | |
| 2024-09-17 12:27:28,001 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: resume | |
| 2024-09-17 12:27:28,001 INFO HandlerThread:4132313 [handler.py:handle_request_resume():719] starting system metrics thread | |
| 2024-09-17 12:27:28,001 INFO HandlerThread:4132313 [system_monitor.py:start():194] Starting system monitor | |
| 2024-09-17 12:27:28,002 INFO SystemMonitor:4132313 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-09-17 12:27:28,005 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started cpu monitoring | |
| 2024-09-17 12:27:28,008 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started disk monitoring | |
| 2024-09-17 12:27:28,008 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started gpu monitoring | |
| 2024-09-17 12:27:28,010 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started memory monitoring | |
| 2024-09-17 12:27:28,011 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started network monitoring | |
| 2024-09-17 12:27:28,066 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry | |
| 2024-09-17 12:27:28,067 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:27:28,204 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_created():271] file/dir created: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/requirements.txt | |
| 2024-09-17 12:27:28,205 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_created():271] file/dir created: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log | |
| 2024-09-17 12:27:28,205 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_created():271] file/dir created: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-metadata.json | |
| 2024-09-17 12:27:30,204 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log | |
| 2024-09-17 12:27:32,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:27:32,935 DEBUG SenderThread:4132313 [sender.py:send():391] send: config | |
| 2024-09-17 12:27:32,937 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry | |
| 2024-09-17 12:27:32,938 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:27:32,938 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry | |
| 2024-09-17 12:27:32,938 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:27:32,938 WARNING SenderThread:4132313 [sender.py:send_metric():1417] Seen metric with glob (shouldn't happen) | |
| 2024-09-17 12:27:32,938 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry | |
| 2024-09-17 12:27:32,938 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry | |
| 2024-09-17 12:27:32,939 DEBUG SenderThread:4132313 [sender.py:send():391] send: config | |
| 2024-09-17 12:27:34,211 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log | |
| 2024-09-17 12:27:37,878 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:27:37,880 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:27:42,877 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:27:42,878 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:27:43,034 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:27:47,879 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:27:48,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:27:54,880 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:27:57,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:27:57,882 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:27:57,922 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:28:00,105 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:28:01,242 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/config.yaml | |
| 2024-09-17 12:28:05,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:28:07,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:28:10,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:28:12,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:28:12,882 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:28:16,038 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:28:17,881 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:28:21,884 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:28:27,884 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:28:27,885 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:28:27,885 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:28:27,886 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:28:28,011 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-09-17 12:28:28,056 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:28:31,893 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:28:31,897 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:28:31,897 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:28:31,898 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:28:31,898 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:28:31,898 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:28:31,898 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:28:31,904 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:28:32,281 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_created():271] file/dir created: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:28:32,889 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:28:33,282 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/config.yaml | |
| 2024-09-17 12:28:37,163 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:28:37,167 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:28:37,168 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:28:37,169 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:28:37,169 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:28:37,169 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:28:37,174 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:28:37,287 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:28:38,159 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:28:38,160 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:28:39,291 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log | |
| 2024-09-17 12:28:42,885 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:28:42,886 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:28:44,160 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:28:48,158 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:28:49,161 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:28:55,161 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:28:57,885 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:28:57,887 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:28:58,095 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:28:58,158 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:29:00,161 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:29:05,170 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:29:06,343 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/config.yaml | |
| 2024-09-17 12:29:08,159 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:29:11,164 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:29:12,889 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:29:12,890 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:29:17,163 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:29:18,158 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:29:22,164 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:29:27,888 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:29:27,889 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:29:28,113 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:29:28,118 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:29:28,172 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:29:33,168 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:29:38,169 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:29:38,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:29:38,188 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:29:38,189 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:29:38,189 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:29:38,196 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:29:38,383 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:29:42,913 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:29:42,914 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:29:43,192 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:29:43,343 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:29:43,345 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:29:43,345 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:29:43,349 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:29:43,389 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:29:45,554 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log | |
| 2024-09-17 12:29:48,168 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:29:49,173 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:29:54,173 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:29:57,893 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:29:57,894 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:29:58,026 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:29:58,169 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:30:00,173 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:30:05,173 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:30:08,169 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:30:11,174 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:30:12,894 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:30:12,894 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:30:16,175 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:30:18,169 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:30:22,174 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:30:27,174 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:30:27,898 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:30:27,899 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:30:28,105 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:30:28,172 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:30:32,176 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:30:37,176 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:30:38,172 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:30:42,898 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:30:42,899 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:30:43,087 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:30:44,559 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:30:44,561 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:30:44,561 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:30:44,568 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:30:44,626 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:30:48,189 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:30:48,214 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:30:49,050 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:30:49,053 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:30:49,053 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:30:49,057 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:30:49,633 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:30:51,690 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log | |
| 2024-09-17 12:30:54,178 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:30:57,898 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:30:57,899 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:30:58,125 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:30:58,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:31:00,179 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:31:06,179 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:31:08,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:31:11,179 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:31:12,899 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:31:12,900 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:31:17,179 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:31:18,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:31:22,180 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:31:27,180 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:31:27,899 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:31:27,900 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:31:28,115 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:31:28,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:31:33,180 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:31:38,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:31:38,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:31:42,546 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:31:42,548 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:31:42,548 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:31:42,553 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:31:42,750 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:31:42,900 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:31:42,901 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:31:43,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:31:47,028 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:31:47,030 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:31:47,030 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:31:47,034 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:31:47,755 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:31:48,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:31:48,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:31:49,759 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log | |
| 2024-09-17 12:31:54,183 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:31:57,903 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:31:57,903 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:31:58,101 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:31:58,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:31:59,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:32:04,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:32:08,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:32:09,184 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:32:12,904 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:32:12,905 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:32:15,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:32:18,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:32:21,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:32:26,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:32:27,904 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:32:27,905 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:32:28,122 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:32:28,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:32:32,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:32:38,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:32:38,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:32:42,904 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:32:42,905 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:32:43,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:32:45,990 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:32:45,992 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:32:45,992 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:32:45,997 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:32:46,827 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:32:48,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:32:48,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:32:50,431 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:32:50,433 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:32:50,433 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:32:50,437 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:32:50,832 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:32:51,907 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log | |
| 2024-09-17 12:32:54,185 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:32:57,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:32:57,905 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:32:58,050 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:32:58,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:32:59,188 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:33:05,186 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:33:08,196 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:33:10,195 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:33:12,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:33:12,905 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:33:16,193 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:33:18,195 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:33:21,193 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:33:26,199 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:33:27,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:33:27,906 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:33:28,135 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:33:28,198 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:33:31,200 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:33:37,198 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:33:38,195 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:33:42,198 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:33:42,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:33:42,906 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:33:48,195 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:33:48,197 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:33:50,740 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:33:50,741 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:33:50,742 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:33:50,745 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:33:50,980 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:33:53,200 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:33:55,232 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:33:55,235 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:33:55,235 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:33:55,239 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:33:55,987 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:33:56,989 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log | |
| 2024-09-17 12:33:57,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:33:57,906 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:33:58,051 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:33:58,195 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:33:59,200 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:34:04,200 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:34:08,209 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:34:09,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:34:12,905 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:34:12,906 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:34:15,208 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:34:18,205 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:34:20,209 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:34:26,209 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:34:27,906 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:34:27,906 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:34:28,050 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:34:28,208 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:34:31,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:34:37,210 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:34:38,207 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:34:42,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:34:42,906 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:34:42,907 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:34:48,207 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:34:48,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:34:50,281 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:34:50,283 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:34:50,283 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:34:50,290 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:34:51,047 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:34:54,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:34:55,374 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:34:55,376 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:34:55,376 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:34:55,381 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:34:56,054 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:34:57,070 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log | |
| 2024-09-17 12:34:57,953 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:34:57,954 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:34:58,118 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:34:58,207 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:35:00,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:35:06,211 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:35:08,213 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:35:11,215 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:35:12,953 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:35:12,954 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:35:16,217 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:35:18,213 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:35:21,217 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:35:26,224 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:35:27,953 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:35:27,954 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:35:28,144 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:35:28,212 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:35:32,223 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:35:37,223 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:35:38,213 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:35:42,225 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:35:42,953 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:35:42,954 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:35:48,213 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:35:48,222 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:35:49,141 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:35:49,143 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:35:49,144 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:35:49,146 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:35:50,122 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:35:53,263 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:35:53,590 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:35:53,592 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:35:53,593 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:35:53,597 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:35:54,127 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:35:55,221 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log | |
| 2024-09-17 12:35:57,954 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:35:57,955 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:35:58,079 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:35:58,220 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:35:59,228 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:36:04,228 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:36:08,219 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:36:09,236 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:36:12,954 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:36:12,955 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:36:15,235 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:36:18,219 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:36:20,241 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:36:25,244 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:36:27,954 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:36:27,955 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:36:28,173 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:36:28,219 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:36:31,242 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:36:36,242 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:36:38,219 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:36:41,243 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:36:42,957 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:36:42,958 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:36:47,089 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:36:47,091 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:36:47,091 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:36:47,092 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:36:47,096 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:36:47,279 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:36:48,220 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:36:51,557 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:36:51,560 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:36:51,560 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:36:51,565 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:36:52,285 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:36:52,566 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:36:53,287 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log | |
| 2024-09-17 12:36:57,956 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:36:57,957 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:36:58,101 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:36:58,107 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:36:58,220 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:37:03,344 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:37:08,220 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:37:08,344 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:37:12,956 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:37:12,957 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:37:14,124 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:37:18,220 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:37:19,349 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:37:24,350 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:37:27,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:37:27,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:37:28,152 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:37:28,221 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:37:29,357 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:37:35,356 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:37:38,221 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:37:40,356 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:37:42,962 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:37:42,963 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:37:45,357 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:37:49,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:37:49,163 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:37:49,167 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:37:49,167 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:37:49,170 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:37:49,351 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:37:50,357 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:37:53,890 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:37:53,892 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:37:53,892 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:37:53,898 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:37:54,358 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:37:55,453 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log | |
| 2024-09-17 12:37:55,795 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:37:57,183 DEBUG SenderThread:4132313 [sender.py:send():391] send: telemetry | |
| 2024-09-17 12:37:57,226 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:37:57,227 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:37:57,231 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:37:57,232 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:37:57,236 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:37:57,237 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:37:57,241 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:37:57,241 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:37:57,245 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:37:57,245 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:37:57,248 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:37:57,249 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:37:57,249 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:37:57,252 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:37:57,455 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:37:57,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:37:57,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:37:58,128 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:37:59,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:38:01,794 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:38:05,499 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:38:05,502 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: partial_history | |
| 2024-09-17 12:38:05,503 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: pause | |
| 2024-09-17 12:38:05,503 INFO HandlerThread:4132313 [handler.py:handle_request_pause():728] stopping system metrics thread | |
| 2024-09-17 12:38:05,504 INFO HandlerThread:4132313 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-09-17 12:38:05,504 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:38:05,506 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-09-17 12:38:05,507 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:38:05,511 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined cpu monitor | |
| 2024-09-17 12:38:05,511 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-09-17 12:38:05,512 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined disk monitor | |
| 2024-09-17 12:38:05,517 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:38:05,517 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:38:05,518 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:38:05,519 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:38:05,519 DEBUG SenderThread:4132313 [sender.py:send():391] send: metric | |
| 2024-09-17 12:38:05,519 DEBUG SenderThread:4132313 [sender.py:send():391] send: history | |
| 2024-09-17 12:38:05,519 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: summary_record | |
| 2024-09-17 12:38:05,522 INFO SenderThread:4132313 [sender.py:_save_file():1466] saving file wandb-summary.json with policy end | |
| 2024-09-17 12:38:05,592 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined gpu monitor | |
| 2024-09-17 12:38:05,593 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined memory monitor | |
| 2024-09-17 12:38:05,593 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined network monitor | |
| 2024-09-17 12:38:05,593 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: resume | |
| 2024-09-17 12:38:05,593 INFO HandlerThread:4132313 [handler.py:handle_request_resume():719] starting system metrics thread | |
| 2024-09-17 12:38:05,593 INFO HandlerThread:4132313 [system_monitor.py:start():194] Starting system monitor | |
| 2024-09-17 12:38:05,594 INFO SystemMonitor:4132313 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-09-17 12:38:05,595 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:38:05,595 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started cpu monitoring | |
| 2024-09-17 12:38:05,597 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started disk monitoring | |
| 2024-09-17 12:38:05,599 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started gpu monitoring | |
| 2024-09-17 12:38:05,602 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started memory monitoring | |
| 2024-09-17 12:38:05,603 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started network monitoring | |
| 2024-09-17 12:38:05,619 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: pause | |
| 2024-09-17 12:38:05,620 INFO HandlerThread:4132313 [handler.py:handle_request_pause():728] stopping system metrics thread | |
| 2024-09-17 12:38:05,620 INFO HandlerThread:4132313 [system_monitor.py:finish():203] Stopping system monitor | |
| 2024-09-17 12:38:05,621 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-09-17 12:38:05,621 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined cpu monitor | |
| 2024-09-17 12:38:05,621 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2024-09-17 12:38:05,622 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined disk monitor | |
| 2024-09-17 12:38:05,622 DEBUG SystemMonitor:4132313 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2024-09-17 12:38:05,679 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined gpu monitor | |
| 2024-09-17 12:38:05,680 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined memory monitor | |
| 2024-09-17 12:38:05,680 INFO HandlerThread:4132313 [interfaces.py:finish():200] Joined network monitor | |
| 2024-09-17 12:38:05,681 DEBUG SenderThread:4132313 [sender.py:send():391] send: stats | |
| 2024-09-17 12:38:06,464 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/wandb-summary.json | |
| 2024-09-17 12:38:07,468 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/output.log | |
| 2024-09-17 12:38:07,683 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:38:09,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:38:12,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:38:12,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:38:12,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:38:17,809 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:38:18,480 INFO Thread-12 :4132313 [dir_watcher.py:_on_file_modified():288] file/dir modified: /nfs/production/literature/amina-mardiyyah/model_save_dir/cl_ct_custom_model/wandb/run-20240917_122726-z0ntw6ob/files/config.yaml | |
| 2024-09-17 12:38:19,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:38:23,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:38:27,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:38:27,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:38:29,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:38:29,152 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:38:34,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:38:39,075 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:38:39,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:38:42,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:38:42,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:38:44,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:38:49,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:38:50,800 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:38:55,801 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:38:57,964 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:38:57,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:38:59,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:39:01,801 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:39:06,801 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:39:09,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:39:12,801 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:39:12,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:39:12,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:39:17,801 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:39:19,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:39:22,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:39:27,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:39:27,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:39:27,965 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:39:29,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:39:33,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:39:38,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:39:39,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:39:42,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:39:42,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:39:44,149 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:39:49,076 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:39:49,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:39:54,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:39:57,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:39:57,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:39:59,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:39:59,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:40:04,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:40:09,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:40:09,802 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:40:12,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:40:12,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:40:14,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:40:19,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:40:19,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:40:24,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:40:27,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:40:27,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:40:29,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:40:29,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:40:34,803 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:40:39,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:40:39,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:40:42,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:40:42,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:40:44,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:40:49,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:40:49,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:40:54,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:40:57,965 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:40:57,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:40:59,077 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:40:59,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:41:04,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:41:09,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:41:10,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:41:12,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:41:12,966 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:41:15,804 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:41:19,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:41:20,805 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:41:26,805 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:41:27,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:41:27,967 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:41:29,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:41:31,805 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:41:36,805 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:41:39,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:41:41,805 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:41:42,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:41:42,967 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:41:47,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:41:49,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:41:52,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:41:57,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:41:57,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:41:57,967 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:41:59,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:42:03,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:42:08,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:42:09,078 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:42:12,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:42:12,967 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:42:14,152 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:42:17,599 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: resume | |
| 2024-09-17 12:42:17,599 INFO HandlerThread:4132313 [handler.py:handle_request_resume():719] starting system metrics thread | |
| 2024-09-17 12:42:17,599 INFO HandlerThread:4132313 [system_monitor.py:start():194] Starting system monitor | |
| 2024-09-17 12:42:17,600 INFO SystemMonitor:4132313 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-09-17 12:42:17,601 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started cpu monitoring | |
| 2024-09-17 12:42:17,605 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started disk monitoring | |
| 2024-09-17 12:42:17,606 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started gpu monitoring | |
| 2024-09-17 12:42:17,608 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started memory monitoring | |
| 2024-09-17 12:42:17,610 INFO SystemMonitor:4132313 [interfaces.py:start():188] Started network monitoring | |
| 2024-09-17 12:42:19,079 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:42:19,806 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:42:24,807 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |
| 2024-09-17 12:42:27,966 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: stop_status | |
| 2024-09-17 12:42:27,967 DEBUG SenderThread:4132313 [sender.py:send_request():418] send_request: stop_status | |
| 2024-09-17 12:42:29,079 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: internal_messages | |
| 2024-09-17 12:42:30,807 DEBUG HandlerThread:4132313 [handler.py:handle_request():158] handle_request: status_report | |