2024-05-25 19:26:38,718 INFO    StreamThr :27237 [internal.py:wandb_internal():85] W&B internal server running at pid: 27237, started at: 2024-05-25 19:26:38.717602
2024-05-25 19:26:38,718 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status
2024-05-25 19:26:38,720 INFO    WriterThread:27237 [datastore.py:open_for_write():87] open: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/run-zf4t8z6a.wandb
2024-05-25 19:26:38,721 DEBUG   SenderThread:27237 [sender.py:send():378] send: header
2024-05-25 19:26:38,721 DEBUG   SenderThread:27237 [sender.py:send():378] send: run
2024-05-25 19:26:39,024 INFO    SenderThread:27237 [dir_watcher.py:__init__():211] watching files in: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files
2024-05-25 19:26:39,024 INFO    SenderThread:27237 [sender.py:_start_run_threads():1123] run started: zf4t8z6a with start time 1716665198.717411
2024-05-25 19:26:39,029 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: check_version
2024-05-25 19:26:39,029 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: check_version
2024-05-25 19:26:39,102 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: run_start
2024-05-25 19:26:39,107 DEBUG   HandlerThread:27237 [system_info.py:__init__():26] System info init
2024-05-25 19:26:39,107 DEBUG   HandlerThread:27237 [system_info.py:__init__():41] System info init done
2024-05-25 19:26:39,107 INFO    HandlerThread:27237 [system_monitor.py:start():194] Starting system monitor
2024-05-25 19:26:39,107 INFO    SystemMonitor:27237 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-05-25 19:26:39,107 INFO    HandlerThread:27237 [system_monitor.py:probe():214] Collecting system info
2024-05-25 19:26:39,108 INFO    SystemMonitor:27237 [interfaces.py:start():188] Started cpu monitoring
2024-05-25 19:26:39,109 INFO    SystemMonitor:27237 [interfaces.py:start():188] Started disk monitoring
2024-05-25 19:26:39,110 INFO    SystemMonitor:27237 [interfaces.py:start():188] Started gpu monitoring
2024-05-25 19:26:39,111 INFO    SystemMonitor:27237 [interfaces.py:start():188] Started memory monitoring
2024-05-25 19:26:39,111 INFO    SystemMonitor:27237 [interfaces.py:start():188] Started network monitoring
2024-05-25 19:26:39,119 DEBUG   HandlerThread:27237 [system_info.py:probe():150] Probing system
2024-05-25 19:26:39,120 DEBUG   HandlerThread:27237 [system_info.py:_probe_git():135] Probing git
2024-05-25 19:26:39,123 DEBUG   HandlerThread:27237 [system_info.py:_probe_git():143] Probing git done
2024-05-25 19:26:39,123 DEBUG   HandlerThread:27237 [system_info.py:probe():198] Probing system done
2024-05-25 19:26:39,123 DEBUG   HandlerThread:27237 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-102-generic-x86_64-with-glibc2.35', 'python': '3.10.14', 'heartbeatAt': '2024-05-25T19:26:39.119542', 'startedAt': '2024-05-25T19:26:38.715184', 'docker': None, 'cuda': None, 'args': ('example/7B.yaml',), 'state': 'running', 'program': '-m train', 'codePathLocal': None, 'git': {'remote': 'https://github.com/mistralai/mistral-finetune.git', 'commit': 'cb892f012ef940bc741dd3b043bc1d06f0cf6085'}, 'email': 'clement.castellon@gmail.com', 'root': '/root/mistral-finetune', 'host': 'scw-tender-archimedes', 'username': 'root', 'executable': '/root/miniconda/envs/chemistralpy310/bin/python', 'cpu_count': 24, 'cpu_count_logical': 24, 'cpu_freq': {'current': 2699.9959999999996, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 902.4293403625488, 'used': 149.24076461791992}}, 'gpu': 'NVIDIA H100 PCIe', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA H100 PCIe', 'memory_total': 85520809984}], 'memory': {'total': 235.9408187866211}}
2024-05-25 19:26:39,123 INFO    HandlerThread:27237 [system_monitor.py:probe():224] Finished collecting system info
2024-05-25 19:26:39,124 INFO    HandlerThread:27237 [system_monitor.py:probe():227] Publishing system info
2024-05-25 19:26:39,124 DEBUG   HandlerThread:27237 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-05-25 19:26:40,025 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_created():271] file/dir created: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/conda-environment.yaml
2024-05-25 19:26:40,039 DEBUG   HandlerThread:27237 [system_info.py:_save_conda():222] Saving conda packages done
2024-05-25 19:26:40,039 INFO    HandlerThread:27237 [system_monitor.py:probe():229] Finished publishing system info
2024-05-25 19:26:40,041 DEBUG   SenderThread:27237 [sender.py:send():378] send: files
2024-05-25 19:26:40,042 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-metadata.json with policy now
2024-05-25 19:26:40,070 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: python_packages
2024-05-25 19:26:40,070 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: python_packages
2024-05-25 19:26:40,070 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:26:40,070 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:26:40,246 DEBUG   SenderThread:27237 [sender.py:send():378] send: telemetry
2024-05-25 19:26:40,496 INFO    wandb-upload_0:27237 [upload_job.py:push():130] Uploaded file /tmp/tmppgb4391mwandb/ms7pal7a-wandb-metadata.json
2024-05-25 19:26:41,025 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_created():271] file/dir created: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:26:41,025 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_created():271] file/dir created: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-metadata.json
2024-05-25 19:26:41,025 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_created():271] file/dir created: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/requirements.txt
2024-05-25 19:26:43,025 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:26:44,451 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:26:45,026 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:26:49,452 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:26:54,452 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:26:55,102 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:26:55,102 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:27:00,327 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:27:05,328 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:27:10,127 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:27:10,127 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:27:11,268 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:27:12,033 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/config.yaml
2024-05-25 19:27:16,400 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:27:21,400 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:27:25,117 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:27:25,117 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:27:27,292 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:27:32,292 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:27:37,293 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:27:39,112 DEBUG   SystemMonitor:27237 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-05-25 19:27:39,113 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:27:40,122 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:27:40,122 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:27:43,266 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:27:48,266 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:27:53,267 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:27:55,133 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:27:55,133 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:27:58,328 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:28:03,329 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:28:08,329 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:28:09,113 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:28:10,166 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:28:10,166 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:28:14,312 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:28:19,312 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:28:24,313 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:28:25,167 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:28:25,167 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:28:29,401 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:28:34,402 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:28:39,114 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:28:40,114 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:28:40,155 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:28:40,155 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:28:45,376 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:28:50,377 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:28:55,170 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:28:55,170 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:28:56,359 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:29:01,360 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:29:06,360 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:29:09,114 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:29:10,163 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:29:10,163 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:29:11,375 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:29:16,376 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:29:21,377 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:29:25,185 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:29:25,185 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:29:27,328 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:29:32,329 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:29:37,330 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:29:39,115 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:29:40,179 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:29:40,179 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:29:42,387 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:29:47,387 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:29:52,388 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:29:55,184 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:29:55,184 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:29:58,364 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:30:03,364 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:30:08,365 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:30:09,115 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:30:10,195 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:30:10,195 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:30:13,366 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:30:18,367 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:30:23,367 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:30:25,231 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:30:25,231 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:30:28,407 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:30:33,408 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:30:38,409 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:30:39,116 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:30:40,200 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:30:40,200 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:30:44,383 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:30:49,383 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:30:54,384 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:30:56,374 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:30:56,374 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:30:59,555 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:31:04,555 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:31:09,116 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:31:10,117 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:31:11,416 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:31:11,416 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:31:15,615 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:31:20,616 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:31:25,616 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:31:26,448 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:31:26,449 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:31:31,592 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:31:36,592 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:31:39,117 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:31:41,422 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:31:41,422 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:31:41,640 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:31:46,640 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:31:51,641 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:31:56,390 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:31:56,390 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:31:57,605 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:32:02,606 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:32:07,607 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:32:09,117 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:32:11,407 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:32:11,407 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:32:13,605 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:32:18,606 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:32:23,607 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:32:26,391 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:32:26,392 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:32:29,534 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:32:34,535 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:32:39,118 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:32:40,118 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:32:41,412 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:32:41,412 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:32:45,627 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:32:50,627 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:32:55,628 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:32:56,449 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:32:56,449 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:33:01,610 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:33:06,610 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:33:09,118 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:33:11,474 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:33:11,475 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:33:11,685 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:33:16,686 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:33:21,686 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:33:26,454 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:33:26,454 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:33:27,651 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:33:32,652 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:33:37,653 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:33:39,118 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:33:41,423 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:33:41,423 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:33:43,550 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:33:48,551 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:33:53,551 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:33:56,443 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:33:56,443 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:33:58,645 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:34:03,645 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:34:08,646 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:34:09,119 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:34:11,433 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:34:11,433 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:34:14,634 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:34:19,634 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:34:24,635 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:34:26,460 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:34:26,460 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:34:29,664 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:34:34,665 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:34:39,119 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:34:40,120 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:34:41,470 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:34:41,470 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:34:45,654 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:34:50,654 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:34:55,655 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:34:56,486 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:34:56,486 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:35:00,711 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:35:05,712 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:35:09,120 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:35:11,120 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:35:11,459 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:35:11,459 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:35:16,591 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:35:21,592 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:35:26,502 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:35:26,503 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:35:26,735 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:35:31,736 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:35:36,737 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:35:39,120 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:35:41,475 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:35:41,475 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:35:42,714 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:35:47,715 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:35:52,716 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:35:56,495 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:35:56,496 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:35:58,652 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:36:03,653 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:36:08,653 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:36:09,121 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:36:11,521 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:36:11,521 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:36:13,763 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:36:18,764 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:36:23,764 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:36:28,082 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:36:28,082 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:36:29,221 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:36:34,221 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:36:39,121 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:36:40,122 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:36:43,093 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:36:43,093 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:36:45,342 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:36:50,343 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:36:55,343 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:36:58,093 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:36:58,093 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:37:01,269 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:37:06,269 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:37:09,122 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:37:12,122 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:37:13,093 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:37:13,093 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:37:17,264 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:37:22,264 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:37:27,265 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:37:28,135 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:37:28,135 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:37:32,284 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:37:37,285 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:37:39,127 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:37:43,098 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:37:43,119 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:37:43,119 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:37:48,289 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:37:53,290 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:37:58,124 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:37:58,124 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:37:58,327 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:38:03,328 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:38:08,328 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:38:09,123 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:38:13,108 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:38:13,109 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:38:14,310 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:38:19,311 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:38:24,311 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:38:28,124 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:38:28,125 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:38:30,302 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:38:35,302 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:38:39,123 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:38:41,124 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:38:43,151 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:38:43,151 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:38:45,208 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:38:46,374 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:38:50,622 DEBUG   SenderThread:27237 [sender.py:send():378] send: telemetry
2024-05-25 19:38:50,622 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:38:51,622 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:38:53,210 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:38:56,366 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:38:56,367 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:38:56,367 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:38:56,368 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:38:57,211 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:38:57,211 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_created():271] file/dir created: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:38:57,368 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:38:58,113 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:38:58,114 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:39:02,163 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:39:02,163 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:39:02,163 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:39:02,164 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:39:02,212 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:39:03,164 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:39:03,212 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:39:07,962 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:39:07,963 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:39:07,963 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:39:07,963 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:39:08,213 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:39:08,964 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:39:09,124 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:39:09,214 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:39:13,114 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:39:13,114 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:39:13,783 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:39:13,784 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:39:13,784 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:39:13,784 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:39:14,215 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:39:14,787 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:39:15,215 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:39:15,215 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/config.yaml
2024-05-25 19:39:19,616 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:39:19,616 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:39:19,617 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:39:19,617 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:39:20,216 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:39:20,617 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:39:21,216 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:39:25,468 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:39:25,469 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:39:25,469 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:39:25,469 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:39:26,217 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:39:26,469 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:39:27,218 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:39:28,114 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:39:28,114 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:39:31,319 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:39:31,319 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:39:31,320 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:39:31,320 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:39:32,219 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:39:32,320 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:39:33,219 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:39:37,172 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:39:37,173 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:39:37,173 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:39:37,173 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:39:37,220 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:39:38,173 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:39:39,124 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:39:39,220 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:39:43,043 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:39:43,044 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:39:43,044 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:39:43,044 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:39:43,114 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:39:43,114 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:39:43,221 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:39:43,241 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:39:45,222 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:39:48,241 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:39:48,932 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:39:48,933 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:39:48,933 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:39:48,933 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:39:49,223 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:39:51,224 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:39:53,934 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:39:54,831 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:39:54,831 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:39:54,831 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:39:54,832 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:39:55,225 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:39:57,225 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:39:58,114 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:39:58,114 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:39:59,258 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:40:00,710 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:40:00,710 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:40:00,711 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:40:00,711 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:40:01,226 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:40:03,227 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:40:04,711 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:40:06,589 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:40:06,590 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:40:06,590 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:40:06,590 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:40:07,228 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:40:07,228 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:40:09,125 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:40:10,125 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:40:12,484 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:40:12,484 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:40:12,485 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:40:12,485 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:40:13,114 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:40:13,114 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:40:13,230 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:40:13,230 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:40:15,322 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:40:18,380 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:40:18,380 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:40:18,381 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:40:18,381 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:40:19,231 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:40:19,232 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:40:20,381 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:40:24,287 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:40:24,288 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:40:24,288 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:40:24,288 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:40:25,233 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:40:25,233 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:40:26,289 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:40:28,114 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:40:28,115 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:40:30,186 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:40:30,186 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:40:30,186 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:40:30,187 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:40:30,234 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:40:31,235 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:40:32,187 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:40:36,078 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:40:36,079 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:40:36,079 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:40:36,079 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:40:36,236 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:40:37,236 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:40:38,080 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:40:39,125 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:40:41,978 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:40:41,979 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:40:41,979 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:40:41,980 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:40:42,237 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:40:43,114 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:40:43,115 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:40:43,238 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:40:43,306 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:40:47,874 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:40:47,875 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:40:47,875 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:40:47,875 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:40:48,239 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:40:48,875 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:40:50,240 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:40:53,773 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:40:53,774 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:40:53,774 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:40:53,774 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:40:54,241 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:40:54,774 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:40:56,242 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:40:58,114 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:40:58,115 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:40:59,679 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:40:59,680 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:40:59,680 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:40:59,680 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:41:00,243 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:41:00,680 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:41:02,243 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:41:05,589 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:41:05,589 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:41:05,589 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:41:05,590 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:41:06,244 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:41:06,590 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:41:08,245 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:41:09,126 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:41:11,496 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:41:11,497 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:41:11,497 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:41:11,497 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:41:12,246 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:41:12,498 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:41:13,115 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:41:13,115 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:41:14,246 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:41:17,398 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:41:17,398 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:41:17,398 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:41:17,399 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:41:18,247 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:41:18,399 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:41:20,248 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:41:23,305 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:41:23,305 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:41:23,305 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:41:23,305 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:41:24,249 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:41:24,306 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:41:26,250 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:41:28,115 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:41:28,115 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:41:29,211 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:41:29,212 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:41:29,212 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:41:29,212 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:41:29,251 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:41:30,213 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:41:32,251 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:41:35,119 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:41:35,123 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:41:35,124 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:41:35,124 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:41:35,252 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:41:36,124 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:41:38,253 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:41:39,126 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:41:41,031 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:41:41,032 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:41:41,032 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:41:41,032 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:41:41,254 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:41:42,032 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:41:43,115 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:41:43,115 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:41:44,255 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:41:46,932 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:41:46,933 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:41:46,933 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:41:46,933 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:41:47,256 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:41:47,934 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:41:48,256 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:41:52,831 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:41:52,832 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:41:52,832 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:41:52,833 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:41:53,258 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:41:53,833 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:41:54,258 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:41:58,115 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:41:58,115 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:41:58,739 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:41:58,739 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:41:58,740 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:41:58,740 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:41:59,259 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:41:59,740 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:42:00,259 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:42:04,650 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:42:04,650 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:42:04,651 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:42:04,651 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:42:05,261 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:42:05,651 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:42:06,261 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:42:09,126 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:42:10,556 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:42:10,557 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:42:10,557 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:42:10,557 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:42:11,263 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:42:11,558 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:42:12,263 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:42:13,115 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:42:13,115 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:42:16,451 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:42:16,451 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:42:16,452 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:42:16,452 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:42:17,264 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:42:17,452 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:42:18,265 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:42:22,351 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:42:22,352 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:42:22,352 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:42:22,352 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:42:23,266 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:42:23,353 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:42:24,266 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:42:28,115 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:42:28,116 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:42:28,256 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:42:28,259 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:42:28,260 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:42:28,260 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:42:28,267 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:42:29,260 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:42:30,268 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:42:34,218 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:42:34,219 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:42:34,219 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:42:34,219 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:42:34,269 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:42:35,219 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:42:36,269 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:42:39,127 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:42:40,084 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:42:40,085 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:42:40,085 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:42:40,085 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:42:40,270 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:42:41,085 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:42:42,271 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:42:43,115 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:42:43,116 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:42:46,009 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:42:46,009 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:42:46,009 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:42:46,010 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:42:46,272 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:42:47,010 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:42:48,272 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:42:51,931 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:42:51,931 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:42:51,932 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:42:51,932 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:42:52,273 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:42:52,932 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:42:54,274 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:42:57,830 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:42:57,831 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:42:57,831 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:42:57,831 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:42:58,115 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:42:58,116 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:42:58,266 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:42:58,275 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:43:00,275 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:43:03,266 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:43:03,734 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:43:03,735 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:43:03,735 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:43:03,736 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:43:04,276 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:43:06,277 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:43:08,736 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:43:09,127 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:43:09,639 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:43:09,640 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:43:09,640 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:43:09,640 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:43:10,278 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:43:12,278 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:43:13,116 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:43:13,116 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:43:14,271 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:43:15,550 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:43:15,551 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:43:15,551 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:43:15,551 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:43:16,279 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:43:18,280 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:43:19,552 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:43:21,462 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:43:21,462 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:43:21,463 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:43:21,463 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:43:22,296 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:43:24,297 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:43:25,463 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:43:27,367 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:43:27,368 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:43:27,368 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:43:27,369 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:43:28,116 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:43:28,116 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:43:28,298 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:43:28,298 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:43:31,328 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:43:33,272 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:43:33,273 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:43:33,273 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:43:33,273 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:43:33,299 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:43:34,299 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:43:37,274 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:43:39,128 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:43:39,196 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:43:39,197 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:43:39,197 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:43:39,197 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:43:39,300 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:43:40,300 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:43:43,116 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:43:43,116 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:43:43,336 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:43:45,100 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:43:45,101 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:43:45,101 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:43:45,101 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:43:45,302 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:43:46,302 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:43:49,102 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:43:51,013 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:43:51,014 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:43:51,014 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:43:51,014 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:43:51,303 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:43:52,303 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:43:55,015 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:43:56,925 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:43:56,926 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:43:56,926 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:43:56,926 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:43:57,304 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:43:58,116 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:43:58,116 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:43:58,305 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:44:00,328 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:44:02,860 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:44:02,860 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:44:02,861 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:44:02,861 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:44:03,306 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:44:04,306 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:44:05,862 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:44:08,771 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:44:08,772 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:44:08,772 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:44:08,772 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:44:09,128 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:44:09,307 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:44:10,308 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:44:11,129 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:44:13,116 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:44:13,116 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:44:14,696 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:44:14,697 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:44:14,697 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:44:14,697 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:44:15,309 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:44:16,309 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:44:16,698 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:44:20,606 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:44:20,607 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:44:20,607 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:44:20,607 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:44:21,310 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:44:22,311 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:44:22,608 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:44:26,537 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:44:26,537 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:44:26,537 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:44:26,538 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:44:27,312 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:44:28,116 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:44:28,117 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:44:28,242 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:44:28,312 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:44:32,435 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:44:32,436 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:44:32,436 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:44:32,436 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:44:33,313 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:44:33,437 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:44:34,314 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:44:38,341 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:44:38,342 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:44:38,342 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:44:38,342 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:44:39,134 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:44:39,135 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:44:39,315 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:44:40,315 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:44:43,116 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:44:43,117 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:44:44,260 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:44:44,260 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:44:44,260 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:44:44,261 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:44:44,261 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:44:44,316 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:44:46,317 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:44:49,262 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:44:50,192 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:44:50,193 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:44:50,193 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:44:50,193 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:44:50,318 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:44:52,318 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:44:55,194 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:44:56,099 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:44:56,099 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:44:56,100 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:44:56,100 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:44:56,319 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:44:58,116 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:44:58,117 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:44:58,320 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:45:00,290 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:45:02,006 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:45:02,007 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:45:02,007 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:45:02,007 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:45:02,321 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:45:04,321 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:45:06,008 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:45:07,910 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:45:07,910 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:45:07,911 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:45:07,911 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:45:08,322 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:45:09,130 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:45:10,323 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:45:11,130 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:45:13,117 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:45:13,117 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:45:13,810 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:45:13,811 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:45:13,811 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:45:13,812 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:45:14,324 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:45:16,324 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:45:16,812 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:45:19,729 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:45:19,730 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:45:19,730 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:45:19,730 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:45:20,325 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:45:22,326 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:45:22,731 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:45:25,632 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:45:25,633 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:45:25,633 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:45:25,633 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:45:26,326 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:45:28,117 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:45:28,117 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:45:28,248 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:45:28,327 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:45:31,538 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:45:31,539 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:45:31,539 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:45:31,539 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:45:32,328 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:45:32,328 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:45:33,540 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:45:37,429 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:45:37,429 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:45:37,429 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:45:37,430 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:45:38,329 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:45:38,330 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:45:39,130 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:45:39,130 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:45:43,117 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:45:43,117 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:45:43,323 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:45:43,324 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:45:43,324 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:45:43,325 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:45:43,331 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:45:44,325 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:45:44,331 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:45:49,230 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:45:49,231 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:45:49,231 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:45:49,231 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:45:49,332 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:45:50,232 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:45:50,332 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:45:55,121 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:45:55,122 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:45:55,122 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:45:55,122 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:45:55,333 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:45:56,123 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:45:56,333 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:45:58,117 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:45:58,117 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:46:01,033 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:46:01,034 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:46:01,034 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:46:01,034 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:46:01,335 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:46:02,035 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:46:02,335 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:46:06,926 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:46:06,927 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:46:06,927 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:46:06,927 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:46:07,336 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:46:07,928 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:46:08,336 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:46:09,131 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:46:12,814 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:46:12,815 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:46:12,815 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:46:12,815 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:46:13,117 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:46:13,117 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:46:13,257 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:46:13,337 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:46:14,338 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:46:18,257 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:46:18,713 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:46:18,713 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:46:18,714 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:46:18,714 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:46:19,339 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:46:20,339 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:46:23,715 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:46:24,622 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:46:24,622 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:46:24,622 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:46:24,622 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:46:25,340 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:46:26,340 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:46:28,117 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:46:28,118 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:46:29,356 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:46:30,530 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:46:30,531 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:46:30,531 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:46:30,531 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:46:31,342 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:46:32,342 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:46:34,532 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:46:36,415 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:46:36,416 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:46:36,416 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:46:36,416 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:46:37,343 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:46:38,343 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:46:39,131 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:46:40,132 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:46:42,304 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:46:42,304 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:46:42,304 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:46:42,305 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:46:42,345 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:46:43,117 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:46:43,118 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:46:44,345 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:46:45,274 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:46:48,198 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:46:48,199 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:46:48,199 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:46:48,200 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:46:48,346 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:46:50,347 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:46:51,200 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:46:54,092 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:46:54,092 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:46:54,092 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:46:54,093 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:46:54,348 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:46:56,348 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:46:57,093 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:46:58,117 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:46:58,118 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:46:59,978 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:46:59,979 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:46:59,979 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:46:59,979 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:47:00,349 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:47:02,350 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:47:02,980 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:47:05,872 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:47:05,872 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:47:05,872 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:47:05,873 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:47:06,350 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:47:08,351 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:47:08,873 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:47:09,131 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:47:11,761 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:47:11,761 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:47:11,762 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:47:11,762 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:47:12,352 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:47:13,118 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:47:13,118 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:47:14,307 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:47:14,352 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:47:17,644 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:47:17,645 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:47:17,645 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:47:17,645 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:47:18,353 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:47:18,354 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:47:19,645 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:47:23,531 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:47:23,532 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:47:23,532 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:47:23,532 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:47:24,355 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:47:24,355 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:47:25,533 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:47:28,118 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:47:28,118 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:47:29,425 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:47:29,426 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:47:29,426 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:47:29,426 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:47:30,356 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:47:30,356 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:47:31,426 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:47:35,345 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:47:35,345 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:47:35,345 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:47:35,346 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:47:35,357 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:47:36,358 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:47:37,346 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:47:39,132 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:47:41,229 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:47:41,229 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:47:41,230 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:47:41,230 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:47:41,359 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:47:42,359 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:47:43,118 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:47:43,118 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:47:43,244 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:47:47,149 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:47:47,153 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:47:47,153 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:47:47,154 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:47:47,360 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:47:48,360 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:47:49,155 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:47:53,038 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:47:53,038 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:47:53,039 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:47:53,039 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:47:53,362 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:47:54,362 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:47:55,039 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:47:58,118 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:47:58,118 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:47:58,942 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:47:58,943 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:47:58,943 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:47:58,943 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:47:59,363 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:48:00,363 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:48:00,944 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:48:04,836 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:48:04,837 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:48:04,837 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:48:04,837 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:48:05,365 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:48:06,365 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:48:06,837 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:48:09,133 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:48:10,733 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:48:10,734 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:48:10,734 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:48:10,734 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:48:11,366 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:48:12,366 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:48:12,735 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:48:13,118 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:48:13,118 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:48:16,619 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:48:16,620 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:48:16,620 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:48:16,620 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:48:17,368 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:48:18,368 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:48:18,621 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:48:22,512 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:48:22,512 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:48:22,513 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:48:22,513 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:48:23,369 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:48:24,370 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:48:24,514 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:48:28,118 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:48:28,119 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:48:28,403 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:48:28,404 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:48:28,404 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:48:28,404 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:48:29,371 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:48:30,371 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:48:30,405 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:48:34,279 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:48:34,280 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:48:34,280 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:48:34,280 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:48:34,372 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:48:35,694 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:48:36,373 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:48:39,133 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:48:40,558 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:48:40,558 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:48:40,558 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:48:40,559 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:48:41,374 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:48:41,559 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:48:42,375 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:48:43,118 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:48:43,119 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:48:46,462 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:48:46,463 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:48:46,463 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:48:46,463 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:48:47,376 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:48:47,464 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:48:48,376 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:48:52,337 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:48:52,338 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:48:52,338 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:48:52,338 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:48:52,377 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:48:53,339 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:48:54,378 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:48:58,118 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:48:58,119 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:48:58,219 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:48:58,329 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:48:58,329 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:48:58,329 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:48:58,379 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:48:59,330 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:49:00,379 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:49:04,099 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:49:04,100 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:49:04,100 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:49:04,100 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:49:04,380 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:49:05,100 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:49:06,381 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:49:09,134 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:49:09,981 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:49:09,982 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:49:09,982 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:49:09,982 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:49:10,382 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:49:10,983 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:49:12,383 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:49:13,118 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:49:13,119 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:49:15,871 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:49:15,871 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:49:15,872 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:49:15,872 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:49:16,384 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:49:16,872 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:49:18,384 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:49:21,767 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:49:21,767 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:49:21,768 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:49:21,768 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:49:22,385 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:49:22,385 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:49:22,768 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:49:27,769 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:49:32,152 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:49:32,153 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:49:33,325 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:49:37,698 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:49:37,698 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:49:37,699 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:49:37,699 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:49:38,389 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:49:38,389 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:49:38,699 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:49:39,134 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:49:43,516 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:49:43,516 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:49:43,517 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:49:43,517 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:49:44,391 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:49:44,391 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:49:44,517 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:49:47,152 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:49:47,153 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:49:49,422 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:49:49,422 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:49:49,423 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:49:49,423 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:49:50,392 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:49:50,392 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:49:50,424 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:49:55,255 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:49:55,256 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:49:55,256 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:49:55,256 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:49:55,393 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:49:56,257 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:49:56,394 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:50:01,151 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:50:01,152 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:50:01,152 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:50:01,152 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:50:01,395 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:50:02,153 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:50:02,153 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:50:02,153 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:50:02,395 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:50:07,078 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:50:07,078 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:50:07,079 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:50:07,079 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:50:07,397 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:50:08,079 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:50:08,397 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:50:09,135 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:50:12,925 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:50:12,925 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:50:12,926 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:50:12,926 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:50:13,398 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:50:13,926 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:50:14,399 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:50:17,152 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:50:17,153 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:50:18,866 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:50:18,866 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:50:18,867 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:50:18,867 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:50:19,400 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:50:19,867 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:50:20,400 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:50:24,758 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:50:24,759 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:50:24,759 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:50:24,759 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:50:25,402 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:50:25,760 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:50:26,402 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:50:30,664 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:50:30,665 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:50:30,665 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:50:30,666 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:50:31,403 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:50:31,666 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:50:32,153 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:50:32,153 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:50:32,403 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:50:36,606 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:50:36,606 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:50:36,607 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:50:36,607 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:50:37,405 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:50:37,607 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:50:38,405 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:50:39,135 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:50:42,462 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:50:42,463 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:50:42,463 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:50:42,463 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:50:43,407 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:50:43,464 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:50:44,407 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:50:47,153 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:50:47,153 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:50:48,406 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:50:48,406 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:50:48,407 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:50:48,407 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:50:48,408 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:50:49,407 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:50:50,409 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:50:54,266 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:50:54,267 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:50:54,267 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:50:54,267 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:50:54,410 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:50:55,268 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:50:56,410 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:51:00,193 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:51:00,199 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:51:00,199 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:51:00,200 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:51:00,412 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:51:01,200 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:51:02,153 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:51:02,153 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:51:02,413 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:51:06,142 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:51:06,142 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:51:06,143 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:51:06,143 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:51:06,414 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:51:07,143 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:51:08,414 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:51:09,136 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:51:11,992 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:51:11,992 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:51:11,993 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:51:11,993 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:51:12,415 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:51:12,993 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:51:14,416 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:51:17,153 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:51:17,154 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:51:17,938 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:51:17,938 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:51:17,939 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:51:17,939 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:51:18,417 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:51:18,417 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:51:18,939 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:51:23,813 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:51:23,814 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:51:23,814 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:51:23,815 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:51:24,418 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:51:24,419 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:51:24,815 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:51:29,716 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:51:29,716 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:51:29,717 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:51:29,717 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:51:30,420 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:51:30,420 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:51:30,717 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:51:32,153 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:51:32,154 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:51:35,632 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:51:35,632 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:51:35,633 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:51:35,633 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:51:36,421 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:51:36,421 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:51:36,633 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:51:39,136 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:51:41,545 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:51:41,545 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:51:41,546 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:51:41,546 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:51:42,423 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:51:42,423 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:51:42,546 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:51:47,153 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:51:47,154 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:51:47,478 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:51:47,478 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:51:47,479 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:51:47,479 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:51:48,424 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:51:48,424 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:51:48,479 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:51:53,354 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:51:53,354 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:51:53,355 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:51:53,355 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:51:53,425 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:51:54,355 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:51:54,425 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:51:59,259 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:51:59,259 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:51:59,260 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:51:59,260 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:51:59,426 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:52:00,260 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:52:00,427 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:52:02,154 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:52:02,154 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:52:05,206 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:52:05,207 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:52:05,207 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:52:05,207 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:52:05,428 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:52:06,207 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:52:06,428 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:52:09,137 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:52:11,070 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:52:11,071 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:52:11,071 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:52:11,071 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:52:11,429 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:52:12,071 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:52:12,430 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:52:16,974 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:52:16,975 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:52:16,975 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:52:16,975 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:52:17,154 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:52:17,154 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:52:17,154 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:52:17,431 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:52:18,431 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:52:22,334 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:52:22,914 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:52:22,914 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:52:22,915 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:52:22,915 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:52:23,432 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:52:24,432 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:52:27,916 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:52:28,802 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:52:28,803 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:52:28,803 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:52:28,803 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:52:29,434 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:52:30,434 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:52:32,154 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:52:32,154 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:52:33,357 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:52:34,750 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:52:34,750 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:52:34,751 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:52:34,751 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:52:35,435 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:52:36,436 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:52:38,752 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:52:39,137 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:52:40,614 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:52:40,615 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:52:40,615 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:52:40,615 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:52:41,437 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:52:42,437 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:52:44,616 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:52:46,517 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:52:46,518 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:52:46,518 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:52:46,518 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:52:47,154 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:52:47,155 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:52:47,438 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:52:48,438 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:52:50,320 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:52:52,470 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:52:52,470 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:52:52,471 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:52:52,471 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:52:53,440 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:52:54,440 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:52:55,471 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:52:58,325 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:52:58,325 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:52:58,325 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:52:58,326 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:52:58,441 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:53:00,441 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:53:01,326 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:53:02,154 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:53:02,155 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:53:04,278 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:53:04,279 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:53:04,279 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:53:04,279 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:53:04,442 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:53:06,443 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:53:07,280 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:53:09,138 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:53:10,150 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:53:10,151 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:53:10,151 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:53:10,151 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:53:10,444 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:53:12,444 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:53:13,152 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:53:16,043 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:53:16,044 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:53:16,044 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:53:16,044 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:53:16,445 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:53:16,445 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:53:17,154 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:53:17,155 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:53:18,306 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:53:21,986 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:53:21,987 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:53:21,987 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:53:21,987 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:53:22,447 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:53:22,447 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:53:23,987 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:53:27,859 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:53:27,860 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:53:27,860 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:53:27,861 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:53:28,448 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:53:28,448 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:53:29,862 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:53:32,155 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:53:32,155 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:53:33,810 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:53:33,810 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:53:33,811 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:53:33,811 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:53:34,450 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:53:34,450 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:53:35,811 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:53:39,138 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:53:39,669 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:53:39,670 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:53:39,670 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:53:39,670 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:53:40,451 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:53:40,452 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:53:41,670 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:53:45,577 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:53:45,578 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:53:45,578 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:53:45,578 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:53:46,453 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:53:46,453 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:53:47,155 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:53:47,155 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:53:47,155 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:53:51,526 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:53:51,526 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:53:51,527 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:53:51,527 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:53:52,455 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:53:52,455 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:53:52,527 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:53:57,387 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:53:57,388 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:53:57,388 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:53:57,388 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:53:57,456 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:53:58,389 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:53:58,456 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:54:02,155 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:54:02,155 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:54:03,350 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:54:03,350 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:54:03,351 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:54:03,351 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:54:03,457 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:54:04,351 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:54:04,458 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:54:09,143 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:54:09,208 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:54:09,209 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:54:09,209 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:54:09,209 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:54:09,459 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:54:10,209 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:54:10,459 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:54:15,101 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:54:15,102 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:54:15,102 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:54:15,102 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:54:15,461 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:54:16,103 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:54:16,461 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:54:17,155 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:54:17,155 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:54:21,046 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:54:21,046 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:54:21,047 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:54:21,047 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:54:21,462 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:54:22,047 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:54:22,463 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:54:26,911 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:54:26,911 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:54:26,912 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:54:26,912 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:54:27,464 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:54:27,912 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:54:28,464 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:54:32,155 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:54:32,155 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:54:32,854 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:54:32,855 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:54:32,855 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:54:32,855 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:54:33,466 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:54:33,855 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:54:34,466 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:54:38,717 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:54:38,718 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:54:38,718 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:54:38,718 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:54:39,139 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:54:39,140 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:54:39,467 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:54:40,468 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:54:44,140 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:54:44,619 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:54:44,619 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:54:44,620 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:54:44,620 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:54:45,469 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:54:46,469 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:54:47,155 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:54:47,156 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:54:49,381 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:54:50,566 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:54:50,566 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:54:50,567 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:54:50,567 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:54:51,471 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:54:52,471 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:54:54,568 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:54:56,426 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:54:56,426 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:54:56,427 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:54:56,427 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:54:56,472 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:54:58,472 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:55:00,427 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:55:02,155 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:55:02,156 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:55:02,382 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:55:02,382 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:55:02,383 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:55:02,383 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:55:02,473 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:55:04,474 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:55:06,384 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:55:08,240 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:55:08,241 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:55:08,241 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:55:08,241 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:55:08,475 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:55:08,475 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:55:09,140 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:55:12,140 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:55:14,149 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:55:14,150 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:55:14,150 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:55:14,150 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:55:14,477 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:55:14,477 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:55:17,151 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:55:17,155 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:55:17,155 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:55:20,082 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:55:20,083 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:55:20,083 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:55:20,083 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:55:20,478 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:55:20,478 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:55:23,084 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:55:25,941 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:55:25,941 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:55:25,941 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:55:25,942 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:55:26,480 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:55:26,480 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:55:28,942 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:55:31,845 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:55:31,846 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:55:31,846 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:55:31,847 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:55:32,156 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:55:32,156 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:55:32,481 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:55:32,481 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:55:34,383 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:55:37,747 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:55:37,747 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:55:37,747 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:55:37,748 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:55:38,483 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:55:38,483 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:55:39,140 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:55:40,141 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:55:43,665 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:55:43,666 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:55:43,666 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:55:43,666 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:55:44,484 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:55:44,484 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:55:45,667 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:55:47,156 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:55:47,156 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:55:49,614 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:55:49,615 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:55:49,615 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:55:49,615 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:55:50,486 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:55:50,486 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:55:51,615 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:55:55,478 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:55:55,479 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:55:55,479 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:55:55,479 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:55:55,487 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:55:56,487 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:55:57,480 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:56:01,381 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:56:01,382 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:56:01,382 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:56:01,382 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:56:01,488 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:56:02,156 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:56:02,156 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:56:02,489 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:56:03,289 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:56:07,334 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:56:07,334 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:56:07,335 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:56:07,335 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:56:07,490 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:56:08,335 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:56:08,490 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:56:09,141 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:56:13,193 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:56:13,194 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:56:13,194 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:56:13,194 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:56:13,491 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:56:14,194 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:56:14,492 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:56:17,156 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:56:17,156 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:56:19,162 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:56:19,162 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:56:19,163 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:56:19,163 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:56:19,493 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:56:20,163 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:56:20,493 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:56:25,025 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:56:25,026 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:56:25,026 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:56:25,026 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:56:25,495 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:56:26,027 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:56:26,495 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:56:30,940 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:56:30,940 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:56:30,940 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:56:30,940 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:56:31,496 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:56:31,941 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:56:32,156 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:56:32,156 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:56:32,496 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:56:36,890 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:56:36,890 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:56:36,891 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:56:36,891 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:56:37,498 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:56:37,891 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:56:38,498 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:56:39,141 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:56:42,746 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:56:42,747 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:56:42,747 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:56:42,747 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:56:43,499 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:56:43,747 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:56:44,500 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:56:47,156 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:56:47,156 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:56:48,686 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:56:48,686 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:56:48,687 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:56:48,687 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:56:49,501 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:56:49,687 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:56:50,501 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:56:54,548 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:56:54,548 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:56:54,548 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:56:54,549 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:56:55,502 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:56:55,549 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:56:56,503 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:57:00,448 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:57:00,449 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:57:00,449 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:57:00,449 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:57:00,504 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:57:01,449 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:57:02,156 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:57:02,157 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:57:02,504 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:57:06,406 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:57:06,406 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:57:06,407 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:57:06,407 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:57:06,505 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:57:06,505 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:57:07,407 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:57:09,142 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:57:12,272 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:57:12,276 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:57:12,276 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:57:12,276 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:57:12,507 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:57:12,507 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:57:13,276 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:57:17,156 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:57:17,157 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:57:18,226 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:57:18,226 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:57:18,227 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:57:18,227 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:57:18,508 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:57:18,509 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:57:19,227 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:57:24,089 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:57:24,090 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:57:24,090 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:57:24,090 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:57:24,510 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:57:24,510 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:57:25,090 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:57:29,989 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:57:29,989 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:57:29,990 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:57:29,990 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:57:30,512 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:57:30,512 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:57:30,990 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:57:32,156 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:57:32,157 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:57:35,930 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:57:35,931 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:57:35,931 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:57:35,932 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:57:36,513 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:57:36,513 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:57:36,932 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:57:39,142 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:57:41,801 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:57:41,802 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:57:41,802 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:57:41,802 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:57:42,515 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:57:42,515 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:57:42,803 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:57:47,156 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:57:47,157 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:57:47,750 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:57:47,750 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:57:47,751 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:57:47,751 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:57:48,516 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:57:48,517 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:57:48,751 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:57:53,626 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:57:53,627 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:57:53,627 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:57:53,627 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:57:54,518 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:57:54,627 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:57:55,518 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:57:59,603 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:57:59,604 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:57:59,604 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:57:59,604 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:58:00,520 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:58:00,604 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:58:01,520 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:58:02,156 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:58:02,157 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:58:05,558 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:58:05,558 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:58:05,559 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:58:05,559 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:58:06,521 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:58:06,559 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:58:07,522 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:58:09,143 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:58:11,411 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:58:11,412 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:58:11,412 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:58:11,412 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:58:11,523 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:58:12,412 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:58:13,523 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:58:17,157 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:58:17,157 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:58:17,326 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:58:17,327 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:58:17,327 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:58:17,327 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:58:17,524 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:58:18,328 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:58:19,525 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:58:23,228 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:58:23,228 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:58:23,229 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:58:23,229 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:58:23,526 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:58:24,229 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:58:25,527 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:58:29,123 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:58:29,123 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:58:29,124 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:58:29,124 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:58:29,528 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:58:30,124 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:58:31,528 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:58:32,157 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:58:32,157 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:58:35,082 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:58:35,082 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:58:35,083 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:58:35,083 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:58:35,336 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:58:35,529 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:58:37,530 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:58:39,143 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:58:41,144 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:58:41,231 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:58:41,231 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:58:41,232 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:58:41,232 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:58:41,531 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:58:43,532 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:58:46,232 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:58:47,125 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:58:47,126 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:58:47,126 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:58:47,126 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:58:47,169 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:58:47,169 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:58:47,533 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:58:49,533 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:58:51,390 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:58:53,031 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:58:53,032 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:58:53,032 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:58:53,032 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:58:53,535 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:58:55,535 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:58:57,033 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:58:58,927 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:58:58,928 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:58:58,928 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:58:58,928 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:58:59,536 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:59:01,537 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:59:02,157 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:59:02,157 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:59:02,158 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:59:04,886 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:59:04,886 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:59:04,887 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:59:04,887 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:59:05,538 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:59:07,538 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:59:07,887 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:59:09,144 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:59:10,749 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:59:10,749 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:59:10,749 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:59:10,750 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:59:11,540 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:59:13,540 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:59:13,750 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:59:16,650 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:59:16,651 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:59:16,651 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:59:16,651 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:59:17,157 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:59:17,157 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:59:17,541 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:59:19,352 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:59:19,542 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:59:22,545 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:59:22,546 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:59:22,546 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:59:22,546 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:59:23,543 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:59:23,543 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:59:24,547 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:59:28,447 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:59:28,447 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:59:28,448 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:59:28,448 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:59:28,544 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:59:29,545 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:59:30,448 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:59:32,157 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:59:32,158 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:59:34,410 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:59:34,410 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:59:34,411 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:59:34,411 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:59:34,546 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:59:35,546 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:59:36,411 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:59:39,144 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 19:59:40,275 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:59:40,276 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:59:40,276 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:59:40,276 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:59:40,548 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:59:41,548 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:59:42,276 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:59:46,172 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:59:46,172 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:59:46,172 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:59:46,173 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:59:46,549 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:59:47,158 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 19:59:47,158 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 19:59:47,358 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:59:47,549 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:59:52,110 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:59:52,110 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:59:52,111 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:59:52,111 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:59:52,551 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:59:53,111 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:59:53,551 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 19:59:57,965 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 19:59:57,965 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 19:59:57,965 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 19:59:57,966 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 19:59:58,552 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 19:59:58,966 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 19:59:59,552 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:00:02,158 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:00:02,158 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:00:03,914 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:00:03,914 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:00:03,915 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:00:03,915 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:00:04,554 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:00:04,915 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:00:05,554 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:00:09,145 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:00:09,782 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:00:09,783 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:00:09,783 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:00:09,783 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:00:10,555 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:00:10,784 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:00:11,556 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:00:15,678 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:00:15,682 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:00:15,682 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:00:15,682 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:00:16,557 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:00:16,683 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:00:17,158 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:00:17,158 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:00:17,557 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:00:21,630 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:00:21,630 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:00:21,631 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:00:21,631 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:00:22,559 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:00:22,631 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:00:23,559 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:00:27,495 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:00:27,496 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:00:27,496 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:00:27,496 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:00:27,560 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:00:28,497 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:00:29,561 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:00:32,158 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:00:32,158 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:00:33,442 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:00:33,442 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:00:33,443 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:00:33,443 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:00:33,562 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:00:34,443 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:00:35,562 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:00:39,145 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:00:39,305 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:00:39,305 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:00:39,306 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:00:39,306 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:00:39,563 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:00:40,306 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:00:41,564 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:00:45,205 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:00:45,205 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:00:45,205 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:00:45,206 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:00:45,565 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:00:46,206 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:00:47,158 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:00:47,158 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:00:47,566 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:00:51,154 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:00:51,154 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:00:51,155 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:00:51,155 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:00:51,567 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:00:52,155 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:00:53,567 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:00:57,023 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:00:57,023 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:00:57,024 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:00:57,024 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:00:57,569 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:00:58,024 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:00:59,569 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:01:02,158 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:01:02,158 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:01:02,974 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:01:02,974 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:01:02,975 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:01:02,975 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:01:03,570 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:01:03,975 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:01:05,571 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:01:08,843 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:01:08,844 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:01:08,844 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:01:08,844 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:01:09,146 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:01:09,147 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:01:09,572 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:01:11,573 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:01:14,147 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:01:14,745 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:01:14,745 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:01:14,745 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:01:14,746 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:01:15,574 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:01:17,158 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:01:17,158 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:01:17,574 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:01:19,373 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:01:20,706 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:01:20,706 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:01:20,707 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:01:20,707 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:01:21,576 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:01:21,576 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:01:24,707 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:01:26,564 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:01:26,565 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:01:26,565 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:01:26,565 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:01:26,577 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:01:27,577 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:01:30,566 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:01:35,566 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:01:39,146 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:01:39,455 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:01:39,455 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:01:40,619 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:01:44,994 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:01:44,994 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:01:44,995 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:01:44,995 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:01:45,581 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:01:45,995 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:01:47,582 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:01:50,807 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:01:50,808 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:01:50,808 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:01:50,808 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:01:51,583 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:01:51,809 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:01:53,584 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:01:54,454 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:01:54,455 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:01:56,673 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:01:56,674 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:01:56,674 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:01:56,674 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:01:57,585 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:01:57,585 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:01:57,675 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:02:02,533 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:02:02,533 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:02:02,533 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:02:02,534 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:02:02,586 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:02:03,534 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:02:03,586 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:02:08,406 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:02:08,406 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:02:08,407 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:02:08,407 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:02:08,588 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:02:09,147 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:02:09,147 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:02:09,454 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:02:09,455 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:02:09,588 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:02:14,300 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:02:14,301 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:02:14,301 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:02:14,301 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:02:14,301 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:02:14,589 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:02:15,589 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:02:19,302 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:02:20,194 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:02:20,194 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:02:20,195 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:02:20,195 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:02:20,591 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:02:21,591 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:02:24,454 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:02:24,455 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:02:24,665 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:02:26,092 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:02:26,093 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:02:26,093 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:02:26,093 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:02:26,592 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:02:27,593 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:02:30,094 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:02:31,994 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:02:31,995 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:02:31,995 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:02:31,995 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:02:32,594 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:02:33,594 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:02:35,996 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:02:37,893 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:02:37,894 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:02:37,894 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:02:37,894 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:02:38,596 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:02:39,147 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:02:39,455 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:02:39,455 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:02:39,596 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:02:41,612 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:02:43,802 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:02:43,803 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:02:43,803 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:02:43,804 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:02:44,597 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:02:45,598 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:02:46,804 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:02:49,704 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:02:49,705 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:02:49,705 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:02:49,706 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:02:50,599 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:02:51,599 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:02:52,706 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:02:54,455 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:02:54,455 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:02:55,603 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:02:55,604 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:02:55,604 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:02:55,604 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:02:56,600 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:02:57,601 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:02:58,605 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:03:01,507 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:03:01,508 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:03:01,508 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:03:01,508 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:03:01,602 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:03:03,602 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:03:04,509 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:03:07,414 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:03:07,415 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:03:07,415 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:03:07,415 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:03:07,603 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:03:09,148 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:03:09,455 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:03:09,455 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:03:09,603 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:03:09,628 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:03:13,324 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:03:13,324 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:03:13,325 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:03:13,325 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:03:13,605 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:03:15,325 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:03:15,605 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:03:19,230 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:03:19,230 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:03:19,230 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:03:19,231 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:03:19,607 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:03:21,231 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:03:21,607 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:03:24,455 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:03:24,455 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:03:25,136 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:03:25,136 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:03:25,137 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:03:25,137 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:03:25,608 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:03:27,137 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:03:27,609 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:03:31,034 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:03:31,034 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:03:31,035 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:03:31,035 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:03:31,610 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:03:33,035 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:03:33,610 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:03:36,924 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:03:36,928 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:03:36,928 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:03:36,929 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:03:37,611 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:03:38,929 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:03:39,148 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:03:39,455 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:03:39,455 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:03:39,612 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:03:42,848 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:03:42,848 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:03:42,848 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:03:42,849 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:03:43,613 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:03:43,613 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:03:44,849 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:03:48,746 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:03:48,747 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:03:48,747 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:03:48,747 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:03:49,615 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:03:49,615 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:03:50,748 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:03:54,455 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:03:54,456 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:03:54,653 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:03:54,671 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:03:54,671 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:03:54,671 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:03:55,616 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:03:55,616 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:03:56,672 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:04:00,602 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:04:00,602 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:04:00,603 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:04:00,603 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:04:00,617 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:04:01,618 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:04:02,603 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:04:06,467 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:04:06,468 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:04:06,468 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:04:06,469 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:04:06,619 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:04:07,619 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:04:08,469 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:04:09,149 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:04:09,455 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:04:09,456 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:04:12,374 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:04:12,375 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:04:12,375 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:04:12,376 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:04:12,621 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:04:13,621 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:04:14,376 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:04:18,267 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:04:18,268 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:04:18,268 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:04:18,268 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:04:18,622 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:04:19,623 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:04:20,269 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:04:24,165 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:04:24,166 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:04:24,166 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:04:24,167 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:04:24,455 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:04:24,456 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:04:24,624 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:04:25,619 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:04:25,624 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:04:30,075 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:04:30,075 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:04:30,076 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:04:30,076 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:04:30,625 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:04:31,076 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:04:31,626 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:04:35,974 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:04:35,975 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:04:35,975 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:04:35,975 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:04:36,627 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:04:36,976 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:04:37,627 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:04:39,149 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:04:39,456 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:04:39,456 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:04:41,870 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:04:41,870 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:04:41,871 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:04:41,871 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:04:42,629 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:04:42,871 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:04:43,629 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:04:47,775 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:04:47,775 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:04:47,776 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:04:47,776 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:04:48,630 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:04:48,776 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:04:49,631 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:04:53,686 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:04:53,686 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:04:53,686 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:04:53,687 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:04:54,456 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:04:54,456 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:04:54,632 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:04:54,665 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:04:55,632 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:04:59,588 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:04:59,589 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:04:59,589 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:04:59,589 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:04:59,634 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:05:00,590 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:05:01,634 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:05:05,493 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:05:05,493 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:05:05,494 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:05:05,494 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:05:05,635 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:05:06,494 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:05:07,636 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:05:09,150 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:05:09,456 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:05:09,456 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:05:11,395 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:05:11,396 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:05:11,396 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:05:11,396 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:05:11,637 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:05:12,396 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:05:13,637 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:05:17,307 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:05:17,308 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:05:17,308 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:05:17,308 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:05:17,638 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:05:18,309 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:05:19,639 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:05:23,219 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:05:23,220 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:05:23,220 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:05:23,220 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:05:23,640 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:05:24,220 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:05:24,456 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:05:24,456 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:05:25,640 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:05:29,134 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:05:29,135 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:05:29,135 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:05:29,135 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:05:29,642 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:05:30,136 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:05:31,642 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:05:35,041 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:05:35,042 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:05:35,042 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:05:35,042 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:05:35,643 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:05:36,043 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:05:37,644 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:05:39,150 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:05:39,456 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:05:39,457 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:05:40,955 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:05:40,956 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:05:40,956 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:05:40,956 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:05:41,645 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:05:41,645 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:05:41,957 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:05:46,853 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:05:46,853 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:05:46,854 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:05:46,854 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:05:47,647 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:05:47,647 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:05:47,854 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:05:52,753 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:05:52,754 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:05:52,754 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:05:52,754 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:05:53,649 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:05:53,649 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:05:53,755 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:05:54,456 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:05:54,457 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:05:58,661 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:05:58,661 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:05:58,662 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:05:58,662 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:05:59,650 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:05:59,651 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:05:59,662 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:06:04,569 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:06:04,570 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:06:04,570 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:06:04,570 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:06:04,652 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:06:05,571 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:06:05,652 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:06:09,151 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:06:09,456 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:06:09,457 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:06:10,481 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:06:10,482 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:06:10,482 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:06:10,482 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:06:10,653 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:06:11,482 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:06:11,654 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:06:16,388 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:06:16,388 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:06:16,389 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:06:16,389 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:06:16,655 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:06:17,389 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:06:17,656 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:06:22,309 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:06:22,309 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:06:22,309 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:06:22,310 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:06:22,657 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:06:23,310 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:06:23,657 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:06:24,456 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:06:24,457 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:06:28,218 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:06:28,219 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:06:28,219 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:06:28,219 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:06:28,659 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:06:29,220 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:06:29,659 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:06:34,125 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:06:34,125 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:06:34,125 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:06:34,126 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:06:34,660 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:06:35,126 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:06:35,660 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:06:39,151 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:06:39,456 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:06:39,457 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:06:40,026 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:06:40,030 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:06:40,031 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:06:40,031 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:06:40,662 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:06:41,031 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:06:41,662 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:06:45,933 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:06:45,933 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:06:45,933 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:06:45,934 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:06:46,664 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:06:46,934 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:06:47,664 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:06:51,842 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:06:51,842 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:06:51,842 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:06:51,843 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:06:52,665 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:06:52,843 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:06:53,666 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:06:54,457 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:06:54,457 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:06:57,748 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:06:57,748 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:06:57,749 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:06:57,749 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:06:58,667 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:06:58,749 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:06:59,667 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:07:03,648 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:07:03,649 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:07:03,649 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:07:03,650 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:07:03,668 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:07:04,650 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:07:05,669 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:07:09,152 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:07:09,457 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:07:09,457 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:07:09,571 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:07:09,636 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:07:09,636 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:07:09,637 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:07:09,670 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:07:10,637 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:07:11,671 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:07:15,459 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:07:15,459 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:07:15,459 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:07:15,460 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:07:15,672 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:07:16,460 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:07:17,672 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:07:21,361 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:07:21,362 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:07:21,362 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:07:21,362 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:07:21,673 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:07:22,363 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:07:23,674 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:07:24,457 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:07:24,457 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:07:27,270 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:07:27,270 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:07:27,271 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:07:27,271 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:07:27,675 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:07:28,271 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:07:29,675 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:07:33,177 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:07:33,178 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:07:33,178 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:07:33,178 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:07:33,676 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:07:34,178 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:07:35,677 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:07:39,085 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:07:39,086 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:07:39,086 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:07:39,087 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:07:39,152 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:07:39,458 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:07:39,459 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:07:39,650 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:07:39,678 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:07:39,678 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:07:44,650 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:07:44,985 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:07:44,985 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:07:44,986 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:07:44,986 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:07:45,679 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:07:45,679 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:07:49,987 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:07:50,895 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:07:50,895 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:07:50,896 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:07:50,896 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:07:51,681 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:07:51,681 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:07:54,458 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:07:54,459 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:07:55,625 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:07:56,797 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:07:56,797 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:07:56,798 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:07:56,798 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:07:57,682 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:07:57,683 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:08:00,799 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:08:02,704 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:08:02,704 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:08:02,704 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:08:02,705 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:08:03,684 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:08:03,684 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:08:06,705 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:08:08,625 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:08:08,625 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:08:08,626 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:08:08,626 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:08:08,685 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:08:09,153 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:08:09,458 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:08:09,459 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:08:09,686 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:08:12,695 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:08:14,531 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:08:14,532 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:08:14,532 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:08:14,532 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:08:14,688 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:08:15,687 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:08:18,533 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:08:20,451 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:08:20,452 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:08:20,452 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:08:20,452 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:08:20,689 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:08:21,689 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:08:24,453 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:08:24,458 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:08:24,459 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:08:26,361 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:08:26,362 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:08:26,362 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:08:26,362 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:08:26,690 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:08:27,691 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:08:30,363 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:08:32,298 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:08:32,299 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:08:32,299 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:08:32,299 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:08:32,692 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:08:33,692 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:08:36,300 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:08:38,196 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:08:38,196 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:08:38,197 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:08:38,197 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:08:38,615 DEBUG   SenderThread:27237 [sender.py:send():378] send: telemetry
2024-05-25 20:08:38,615 DEBUG   SenderThread:27237 [sender.py:send():378] send: exit
2024-05-25 20:08:38,615 INFO    SenderThread:27237 [sender.py:send_exit():585] handling exit code: 0
2024-05-25 20:08:38,615 INFO    SenderThread:27237 [sender.py:send_exit():587] handling runtime: 2519
2024-05-25 20:08:38,616 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:08:38,616 INFO    SenderThread:27237 [sender.py:send_exit():593] send defer
2024-05-25 20:08:38,616 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: defer
2024-05-25 20:08:38,616 INFO    HandlerThread:27237 [handler.py:handle_request_defer():184] handle defer: 0
2024-05-25 20:08:38,616 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: defer
2024-05-25 20:08:38,616 INFO    SenderThread:27237 [sender.py:send_request_defer():609] handle sender defer: 0
2024-05-25 20:08:38,616 INFO    SenderThread:27237 [sender.py:transition_state():613] send defer: 1
2024-05-25 20:08:38,616 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: defer
2024-05-25 20:08:38,616 INFO    HandlerThread:27237 [handler.py:handle_request_defer():184] handle defer: 1
2024-05-25 20:08:38,616 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: defer
2024-05-25 20:08:38,616 INFO    SenderThread:27237 [sender.py:send_request_defer():609] handle sender defer: 1
2024-05-25 20:08:38,616 INFO    SenderThread:27237 [sender.py:transition_state():613] send defer: 2
2024-05-25 20:08:38,616 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: defer
2024-05-25 20:08:38,616 INFO    HandlerThread:27237 [handler.py:handle_request_defer():184] handle defer: 2
2024-05-25 20:08:38,616 INFO    HandlerThread:27237 [system_monitor.py:finish():203] Stopping system monitor
2024-05-25 20:08:38,617 INFO    HandlerThread:27237 [interfaces.py:finish():200] Joined cpu monitor
2024-05-25 20:08:38,617 DEBUG   SystemMonitor:27237 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-05-25 20:08:38,617 DEBUG   SystemMonitor:27237 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-05-25 20:08:38,618 INFO    HandlerThread:27237 [interfaces.py:finish():200] Joined disk monitor
2024-05-25 20:08:38,641 INFO    HandlerThread:27237 [interfaces.py:finish():200] Joined gpu monitor
2024-05-25 20:08:38,642 INFO    HandlerThread:27237 [interfaces.py:finish():200] Joined memory monitor
2024-05-25 20:08:38,642 INFO    HandlerThread:27237 [interfaces.py:finish():200] Joined network monitor
2024-05-25 20:08:38,642 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: defer
2024-05-25 20:08:38,642 INFO    SenderThread:27237 [sender.py:send_request_defer():609] handle sender defer: 2
2024-05-25 20:08:38,642 INFO    SenderThread:27237 [sender.py:transition_state():613] send defer: 3
2024-05-25 20:08:38,642 DEBUG   SenderThread:27237 [sender.py:send():378] send: stats
2024-05-25 20:08:38,642 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: defer
2024-05-25 20:08:38,642 INFO    HandlerThread:27237 [handler.py:handle_request_defer():184] handle defer: 3
2024-05-25 20:08:38,643 DEBUG   SenderThread:27237 [sender.py:send():378] send: history
2024-05-25 20:08:38,643 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:08:38,643 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:08:38,643 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: defer
2024-05-25 20:08:38,643 INFO    SenderThread:27237 [sender.py:send_request_defer():609] handle sender defer: 3
2024-05-25 20:08:38,643 INFO    SenderThread:27237 [sender.py:transition_state():613] send defer: 4
2024-05-25 20:08:38,643 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: defer
2024-05-25 20:08:38,643 INFO    HandlerThread:27237 [handler.py:handle_request_defer():184] handle defer: 4
2024-05-25 20:08:38,643 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: defer
2024-05-25 20:08:38,644 INFO    SenderThread:27237 [sender.py:send_request_defer():609] handle sender defer: 4
2024-05-25 20:08:38,644 INFO    SenderThread:27237 [sender.py:transition_state():613] send defer: 5
2024-05-25 20:08:38,644 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: defer
2024-05-25 20:08:38,644 INFO    HandlerThread:27237 [handler.py:handle_request_defer():184] handle defer: 5
2024-05-25 20:08:38,644 DEBUG   SenderThread:27237 [sender.py:send():378] send: summary
2024-05-25 20:08:38,644 INFO    SenderThread:27237 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:08:38,644 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: defer
2024-05-25 20:08:38,644 INFO    SenderThread:27237 [sender.py:send_request_defer():609] handle sender defer: 5
2024-05-25 20:08:38,644 INFO    SenderThread:27237 [sender.py:transition_state():613] send defer: 6
2024-05-25 20:08:38,644 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: defer
2024-05-25 20:08:38,644 INFO    HandlerThread:27237 [handler.py:handle_request_defer():184] handle defer: 6
2024-05-25 20:08:38,644 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: defer
2024-05-25 20:08:38,644 INFO    SenderThread:27237 [sender.py:send_request_defer():609] handle sender defer: 6
2024-05-25 20:08:38,646 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:08:38,694 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:08:38,798 INFO    SenderThread:27237 [sender.py:transition_state():613] send defer: 7
2024-05-25 20:08:38,798 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: defer
2024-05-25 20:08:38,798 INFO    HandlerThread:27237 [handler.py:handle_request_defer():184] handle defer: 7
2024-05-25 20:08:38,798 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: defer
2024-05-25 20:08:38,798 INFO    SenderThread:27237 [sender.py:send_request_defer():609] handle sender defer: 7
2024-05-25 20:08:39,322 INFO    SenderThread:27237 [sender.py:transition_state():613] send defer: 8
2024-05-25 20:08:39,322 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: defer
2024-05-25 20:08:39,322 INFO    HandlerThread:27237 [handler.py:handle_request_defer():184] handle defer: 8
2024-05-25 20:08:39,322 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: defer
2024-05-25 20:08:39,322 INFO    SenderThread:27237 [sender.py:send_request_defer():609] handle sender defer: 8
2024-05-25 20:08:39,322 INFO    SenderThread:27237 [job_builder.py:build():432] Attempting to build job artifact
2024-05-25 20:08:39,323 INFO    SenderThread:27237 [job_builder.py:_get_source_type():565] is repo sourced job
2024-05-25 20:08:39,323 WARNING SenderThread:27237 [job_builder.py:_log_if_verbose():267] No program path found, not creating job artifact. See https://docs.wandb.ai/guides/launch/create-job
2024-05-25 20:08:39,323 INFO    SenderThread:27237 [sender.py:transition_state():613] send defer: 9
2024-05-25 20:08:39,323 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: defer
2024-05-25 20:08:39,323 INFO    HandlerThread:27237 [handler.py:handle_request_defer():184] handle defer: 9
2024-05-25 20:08:39,323 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: defer
2024-05-25 20:08:39,323 INFO    SenderThread:27237 [sender.py:send_request_defer():609] handle sender defer: 9
2024-05-25 20:08:39,323 INFO    SenderThread:27237 [dir_watcher.py:finish():358] shutting down directory watcher
2024-05-25 20:08:39,616 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: poll_exit
2024-05-25 20:08:39,694 INFO    Thread-12 :27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:08:39,694 INFO    SenderThread:27237 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/config.yaml
2024-05-25 20:08:39,695 INFO    SenderThread:27237 [dir_watcher.py:finish():388] scan: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files
2024-05-25 20:08:39,695 INFO    SenderThread:27237 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/conda-environment.yaml conda-environment.yaml
2024-05-25 20:08:39,695 INFO    SenderThread:27237 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log output.log
2024-05-25 20:08:39,695 INFO    SenderThread:27237 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-metadata.json wandb-metadata.json
2024-05-25 20:08:39,695 INFO    SenderThread:27237 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/config.yaml config.yaml
2024-05-25 20:08:39,695 INFO    SenderThread:27237 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/requirements.txt requirements.txt
2024-05-25 20:08:39,695 INFO    SenderThread:27237 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json wandb-summary.json
2024-05-25 20:08:39,699 INFO    SenderThread:27237 [sender.py:transition_state():613] send defer: 10
2024-05-25 20:08:39,699 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: poll_exit
2024-05-25 20:08:39,700 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: defer
2024-05-25 20:08:39,700 INFO    HandlerThread:27237 [handler.py:handle_request_defer():184] handle defer: 10
2024-05-25 20:08:39,701 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: defer
2024-05-25 20:08:39,701 INFO    SenderThread:27237 [sender.py:send_request_defer():609] handle sender defer: 10
2024-05-25 20:08:39,701 INFO    SenderThread:27237 [file_pusher.py:finish():169] shutting down file pusher
2024-05-25 20:08:40,051 INFO    wandb-upload_1:27237 [upload_job.py:push():130] Uploaded file /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/conda-environment.yaml
2024-05-25 20:08:40,069 INFO    wandb-upload_4:27237 [upload_job.py:push():130] Uploaded file /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/wandb-summary.json
2024-05-25 20:08:40,077 INFO    wandb-upload_3:27237 [upload_job.py:push():130] Uploaded file /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/requirements.txt
2024-05-25 20:08:40,085 INFO    wandb-upload_2:27237 [upload_job.py:push():130] Uploaded file /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/config.yaml
2024-05-25 20:08:40,258 INFO    wandb-upload_0:27237 [upload_job.py:push():130] Uploaded file /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/files/output.log
2024-05-25 20:08:40,459 INFO    Thread-11 (_thread_body):27237 [sender.py:transition_state():613] send defer: 11
2024-05-25 20:08:40,459 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: defer
2024-05-25 20:08:40,459 INFO    HandlerThread:27237 [handler.py:handle_request_defer():184] handle defer: 11
2024-05-25 20:08:40,460 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: defer
2024-05-25 20:08:40,460 INFO    SenderThread:27237 [sender.py:send_request_defer():609] handle sender defer: 11
2024-05-25 20:08:40,460 INFO    SenderThread:27237 [file_pusher.py:join():175] waiting for file pusher
2024-05-25 20:08:40,460 INFO    SenderThread:27237 [sender.py:transition_state():613] send defer: 12
2024-05-25 20:08:40,460 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: defer
2024-05-25 20:08:40,460 INFO    HandlerThread:27237 [handler.py:handle_request_defer():184] handle defer: 12
2024-05-25 20:08:40,460 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: defer
2024-05-25 20:08:40,460 INFO    SenderThread:27237 [sender.py:send_request_defer():609] handle sender defer: 12
2024-05-25 20:08:40,460 INFO    SenderThread:27237 [file_stream.py:finish():601] file stream finish called
2024-05-25 20:08:40,602 INFO    SenderThread:27237 [file_stream.py:finish():605] file stream finish is done
2024-05-25 20:08:40,602 INFO    SenderThread:27237 [sender.py:transition_state():613] send defer: 13
2024-05-25 20:08:40,602 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: defer
2024-05-25 20:08:40,602 INFO    HandlerThread:27237 [handler.py:handle_request_defer():184] handle defer: 13
2024-05-25 20:08:40,602 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: defer
2024-05-25 20:08:40,602 INFO    SenderThread:27237 [sender.py:send_request_defer():609] handle sender defer: 13
2024-05-25 20:08:40,602 INFO    SenderThread:27237 [sender.py:transition_state():613] send defer: 14
2024-05-25 20:08:40,602 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: defer
2024-05-25 20:08:40,602 INFO    HandlerThread:27237 [handler.py:handle_request_defer():184] handle defer: 14
2024-05-25 20:08:40,603 DEBUG   SenderThread:27237 [sender.py:send():378] send: final
2024-05-25 20:08:40,603 DEBUG   SenderThread:27237 [sender.py:send():378] send: footer
2024-05-25 20:08:40,603 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: defer
2024-05-25 20:08:40,603 INFO    SenderThread:27237 [sender.py:send_request_defer():609] handle sender defer: 14
2024-05-25 20:08:40,603 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: poll_exit
2024-05-25 20:08:40,604 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: poll_exit
2024-05-25 20:08:40,604 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: internal_messages
2024-05-25 20:08:40,604 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: server_info
2024-05-25 20:08:40,604 DEBUG   SenderThread:27237 [sender.py:send_request():405] send_request: server_info
2024-05-25 20:08:40,646 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: get_summary
2024-05-25 20:08:40,646 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: sampled_history
2024-05-25 20:08:40,736 DEBUG   HandlerThread:27237 [handler.py:handle_request():158] handle_request: shutdown
2024-05-25 20:08:40,736 INFO    HandlerThread:27237 [handler.py:finish():882] shutting down handler
2024-05-25 20:08:41,604 INFO    WriterThread:27237 [datastore.py:close():296] close: /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/run-zf4t8z6a.wandb
2024-05-25 20:08:41,736 INFO    SenderThread:27237 [sender.py:finish():1545] shutting down sender
2024-05-25 20:08:41,736 INFO    SenderThread:27237 [file_pusher.py:finish():169] shutting down file pusher
2024-05-25 20:08:41,736 INFO    SenderThread:27237 [file_pusher.py:join():175] waiting for file pusher