2024-05-25 20:26:13,703 INFO    StreamThr :33547 [internal.py:wandb_internal():85] W&B internal server running at pid: 33547, started at: 2024-05-25 20:26:13.703173
2024-05-25 20:26:13,704 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status
2024-05-25 20:26:13,706 INFO    WriterThread:33547 [datastore.py:open_for_write():87] open: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/run-sapihyp4.wandb
2024-05-25 20:26:13,707 DEBUG   SenderThread:33547 [sender.py:send():378] send: header
2024-05-25 20:26:13,707 DEBUG   SenderThread:33547 [sender.py:send():378] send: run
2024-05-25 20:26:14,049 INFO    SenderThread:33547 [dir_watcher.py:__init__():211] watching files in: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files
2024-05-25 20:26:14,049 INFO    SenderThread:33547 [sender.py:_start_run_threads():1123] run started: sapihyp4 with start time 1716668773.703018
2024-05-25 20:26:14,054 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: check_version
2024-05-25 20:26:14,054 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: check_version
2024-05-25 20:26:14,130 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: run_start
2024-05-25 20:26:14,135 DEBUG   HandlerThread:33547 [system_info.py:__init__():26] System info init
2024-05-25 20:26:14,135 DEBUG   HandlerThread:33547 [system_info.py:__init__():41] System info init done
2024-05-25 20:26:14,135 INFO    HandlerThread:33547 [system_monitor.py:start():194] Starting system monitor
2024-05-25 20:26:14,135 INFO    SystemMonitor:33547 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-05-25 20:26:14,135 INFO    HandlerThread:33547 [system_monitor.py:probe():214] Collecting system info
2024-05-25 20:26:14,136 INFO    SystemMonitor:33547 [interfaces.py:start():188] Started cpu monitoring
2024-05-25 20:26:14,137 INFO    SystemMonitor:33547 [interfaces.py:start():188] Started disk monitoring
2024-05-25 20:26:14,137 INFO    SystemMonitor:33547 [interfaces.py:start():188] Started gpu monitoring
2024-05-25 20:26:14,138 INFO    SystemMonitor:33547 [interfaces.py:start():188] Started memory monitoring
2024-05-25 20:26:14,138 INFO    SystemMonitor:33547 [interfaces.py:start():188] Started network monitoring
2024-05-25 20:26:14,147 DEBUG   HandlerThread:33547 [system_info.py:probe():150] Probing system
2024-05-25 20:26:14,148 DEBUG   HandlerThread:33547 [system_info.py:_probe_git():135] Probing git
2024-05-25 20:26:14,151 DEBUG   HandlerThread:33547 [system_info.py:_probe_git():143] Probing git done
2024-05-25 20:26:14,151 DEBUG   HandlerThread:33547 [system_info.py:probe():198] Probing system done
2024-05-25 20:26:14,151 DEBUG   HandlerThread:33547 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-102-generic-x86_64-with-glibc2.35', 'python': '3.10.14', 'heartbeatAt': '2024-05-25T20:26:14.147426', 'startedAt': '2024-05-25T20:26:13.700754', 'docker': None, 'cuda': None, 'args': ('example/7B.yaml',), 'state': 'running', 'program': '-m train', 'codePathLocal': None, 'git': {'remote': 'https://github.com/mistralai/mistral-finetune.git', 'commit': 'cb892f012ef940bc741dd3b043bc1d06f0cf6085'}, 'email': 'clement.castellon@gmail.com', 'root': '/root/mistral-finetune', 'host': 'scw-tender-archimedes', 'username': 'root', 'executable': '/root/miniconda/envs/chemistralpy310/bin/python', 'cpu_count': 24, 'cpu_count_logical': 24, 'cpu_freq': {'current': 2699.9959999999996, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 902.4293403625488, 'used': 150.20981216430664}}, 'gpu': 'NVIDIA H100 PCIe', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA H100 PCIe', 'memory_total': 85520809984}], 'memory': {'total': 235.9408187866211}}
2024-05-25 20:26:14,151 INFO    HandlerThread:33547 [system_monitor.py:probe():224] Finished collecting system info
2024-05-25 20:26:14,151 INFO    HandlerThread:33547 [system_monitor.py:probe():227] Publishing system info
2024-05-25 20:26:14,151 DEBUG   HandlerThread:33547 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-05-25 20:26:15,050 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_created():271] file/dir created: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/conda-environment.yaml
2024-05-25 20:26:15,117 DEBUG   HandlerThread:33547 [system_info.py:_save_conda():222] Saving conda packages done
2024-05-25 20:26:15,117 INFO    HandlerThread:33547 [system_monitor.py:probe():229] Finished publishing system info
2024-05-25 20:26:15,120 DEBUG   SenderThread:33547 [sender.py:send():378] send: files
2024-05-25 20:26:15,120 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-metadata.json with policy now
2024-05-25 20:26:15,147 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: python_packages
2024-05-25 20:26:15,147 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: python_packages
2024-05-25 20:26:15,147 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:26:15,147 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:26:15,340 DEBUG   SenderThread:33547 [sender.py:send():378] send: telemetry
2024-05-25 20:26:15,598 INFO    wandb-upload_0:33547 [upload_job.py:push():130] Uploaded file /tmp/tmp1tdz824awandb/653o9vjt-wandb-metadata.json
2024-05-25 20:26:16,050 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_created():271] file/dir created: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/requirements.txt
2024-05-25 20:26:16,050 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_created():271] file/dir created: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:26:16,050 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_created():271] file/dir created: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-metadata.json
2024-05-25 20:26:18,050 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:26:19,275 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:26:24,276 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:26:29,277 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:26:30,188 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:26:30,189 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:26:34,415 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:26:39,416 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:26:44,418 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:26:45,057 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/config.yaml
2024-05-25 20:26:45,183 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:26:45,184 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:26:50,402 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:26:55,403 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:27:00,173 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:27:00,174 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:27:00,406 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:27:05,406 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:27:10,407 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:27:14,139 DEBUG   SystemMonitor:33547 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-05-25 20:27:14,140 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:27:15,189 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:27:15,189 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:27:16,347 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:27:21,348 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:27:26,349 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:27:30,210 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:27:30,211 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:27:32,346 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:27:37,347 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:27:42,348 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:27:44,140 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:27:45,211 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:27:45,211 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:27:47,369 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:27:52,369 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:27:57,370 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:28:00,222 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:28:00,222 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:28:02,450 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:28:07,451 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:28:12,451 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:28:14,141 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:28:15,263 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:28:15,263 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:28:17,476 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:28:22,477 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:28:27,478 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:28:30,222 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:28:30,223 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:28:33,464 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:28:38,465 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:28:43,466 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:28:44,141 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:28:45,243 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:28:45,244 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:28:49,425 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:28:54,426 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:28:59,426 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:29:00,249 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:29:00,250 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:29:04,454 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:29:09,455 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:29:14,142 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:29:15,142 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:29:15,229 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:29:15,229 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:29:20,459 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:29:25,460 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:29:30,239 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:29:30,239 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:29:31,396 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:29:36,397 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:29:41,397 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:29:44,142 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:29:45,272 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:29:45,272 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:29:46,448 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:29:51,449 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:29:56,450 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:30:00,315 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:30:00,315 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:30:01,462 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:30:06,463 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:30:11,464 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:30:14,142 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:30:15,249 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:30:15,250 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:30:16,473 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:30:21,474 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:30:26,474 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:30:30,260 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:30:30,260 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:30:32,395 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:30:37,395 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:30:42,396 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:30:44,143 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:30:45,276 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:30:45,277 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:30:47,427 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:30:52,428 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:30:57,428 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:31:00,314 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:31:00,314 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:31:02,519 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:31:07,520 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:31:12,521 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:31:14,144 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:31:15,271 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:31:15,271 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:31:18,513 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:31:23,513 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:31:28,514 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:31:30,277 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:31:30,277 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:31:34,452 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:31:39,453 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:31:44,144 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:31:45,145 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:31:45,277 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:31:45,277 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:31:50,503 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:31:55,503 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:32:00,303 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:32:00,303 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:32:01,493 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:32:06,494 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:32:11,495 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:32:14,144 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:32:15,308 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:32:15,309 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:32:17,488 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:32:22,488 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:32:27,489 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:32:32,490 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:32:32,584 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:32:32,584 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:32:37,730 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:32:42,731 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:32:44,145 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:32:47,578 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:32:47,579 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:32:47,756 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:32:52,757 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:32:57,757 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:33:02,610 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:33:02,611 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:33:03,748 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:33:08,749 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:33:13,749 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:33:14,146 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:33:17,589 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:33:17,589 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:33:18,797 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:33:23,798 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:33:28,798 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:33:32,590 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:33:32,590 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:33:34,764 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:33:39,765 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:33:44,146 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:33:45,147 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:33:47,610 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:33:47,610 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:33:50,764 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:33:55,764 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:34:00,765 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:34:02,623 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:34:02,623 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:34:05,777 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:34:10,778 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:34:14,147 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:34:16,147 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:34:17,605 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:34:17,606 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:34:21,845 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:34:26,845 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:34:31,846 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:34:32,611 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:34:32,611 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:34:36,855 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:34:41,856 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:34:44,147 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:34:47,148 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:34:47,611 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:34:47,611 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:34:52,823 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:34:57,824 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:35:02,616 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:35:02,617 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:35:03,763 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:35:08,764 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:35:13,765 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:35:14,148 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:35:17,622 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:35:17,622 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:35:18,866 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:35:23,867 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:35:28,867 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:35:32,645 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:35:32,645 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:35:34,844 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:35:39,844 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:35:44,148 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:35:45,149 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:35:47,642 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:35:47,643 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:35:50,862 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:35:55,862 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:36:00,863 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:36:02,643 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:36:02,643 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:36:05,864 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:36:10,865 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:36:14,149 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:36:16,149 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:36:17,663 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:36:17,663 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:36:21,866 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:36:26,866 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:36:31,867 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:36:32,680 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:36:32,680 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:36:36,868 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:36:41,869 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:36:44,149 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:36:47,150 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:36:47,659 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:36:47,660 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:36:52,841 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:36:57,842 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:37:02,732 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:37:02,732 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:37:02,936 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:37:07,937 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:37:12,938 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:37:14,153 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:37:17,686 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:37:17,686 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:37:18,925 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:37:23,926 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:37:28,927 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:37:32,726 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:37:32,727 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:37:34,921 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:37:39,922 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:37:44,150 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:37:45,151 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:37:47,690 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:37:47,691 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:37:50,828 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:37:55,829 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:38:00,830 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:38:02,690 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:38:02,691 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:38:05,832 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:38:10,833 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:38:14,151 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:38:16,152 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:38:17,690 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:38:17,691 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:38:21,827 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:38:27,797 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:38:28,229 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:38:32,675 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:38:32,675 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:38:32,849 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:38:33,049 DEBUG   SenderThread:33547 [sender.py:send():378] send: telemetry
2024-05-25 20:38:33,049 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:38:34,231 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:38:38,050 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:38:38,802 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:38:38,803 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:38:38,804 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:38:38,804 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:38:39,232 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_created():271] file/dir created: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:38:40,232 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:38:43,807 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:38:44,151 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:38:44,233 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/config.yaml
2024-05-25 20:38:44,598 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:38:44,599 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:38:44,599 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:38:44,599 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:38:45,234 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:38:46,234 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:38:47,675 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:38:47,676 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:38:49,910 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:38:50,411 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:38:50,411 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:38:50,411 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:38:50,412 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:38:51,235 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:38:52,236 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:38:55,412 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:38:56,232 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:38:56,232 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:38:56,233 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:38:56,233 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:38:56,237 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:38:58,237 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:39:01,234 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:39:02,070 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:39:02,070 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:39:02,071 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:39:02,071 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:39:02,238 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:39:02,676 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:39:02,676 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:39:04,239 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:39:06,909 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:39:07,919 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:39:07,920 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:39:07,920 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:39:07,920 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:39:08,240 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:39:10,240 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:39:11,921 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:39:13,772 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:39:13,773 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:39:13,773 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:39:13,773 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:39:14,152 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:39:14,241 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:39:17,153 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:39:17,243 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:39:17,676 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:39:17,676 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:39:19,639 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:39:19,640 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:39:19,640 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:39:19,640 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:39:20,244 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:39:21,244 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:39:22,641 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:39:25,512 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:39:25,513 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:39:25,513 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:39:25,514 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:39:26,245 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:39:27,246 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:39:28,514 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:39:31,414 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:39:31,414 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:39:31,415 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:39:31,415 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:39:32,247 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:39:32,676 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:39:32,676 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:39:33,247 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:39:33,852 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:39:37,311 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:39:37,312 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:39:37,312 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:39:37,312 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:39:38,248 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:39:39,249 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:39:39,313 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:39:43,196 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:39:43,197 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:39:43,197 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:39:43,197 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:39:43,250 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:39:44,152 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:39:45,153 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:39:45,250 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:39:47,676 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:39:47,676 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:39:49,077 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:39:49,078 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:39:49,078 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:39:49,078 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:39:49,251 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:39:51,079 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:39:51,252 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:39:54,975 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:39:54,975 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:39:54,975 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:39:54,976 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:39:55,253 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:39:56,976 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:39:57,253 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:40:00,873 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:40:00,874 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:40:00,874 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:40:00,875 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:40:01,254 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:40:02,676 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:40:02,676 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:40:02,904 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:40:03,255 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:40:06,774 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:40:06,775 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:40:06,775 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:40:06,775 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:40:07,256 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:40:08,776 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:40:09,256 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:40:12,674 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:40:12,675 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:40:12,675 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:40:12,676 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:40:13,257 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:40:14,153 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:40:14,153 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:40:15,258 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:40:17,676 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:40:17,676 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:40:18,575 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:40:18,575 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:40:18,575 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:40:18,575 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:40:19,259 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:40:19,576 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:40:21,259 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:40:24,474 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:40:24,474 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:40:24,474 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:40:24,475 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:40:25,260 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:40:25,475 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:40:27,261 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:40:30,367 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:40:30,368 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:40:30,368 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:40:30,369 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:40:31,262 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:40:31,369 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:40:32,676 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:40:32,676 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:40:33,262 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:40:36,262 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:40:36,263 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:40:36,263 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:40:36,263 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:40:36,264 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:40:37,264 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:40:39,265 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:40:42,161 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:40:42,162 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:40:42,162 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:40:42,162 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:40:42,266 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:40:43,163 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:40:44,153 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:40:45,266 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:40:47,676 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:40:47,677 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:40:48,073 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:40:48,074 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:40:48,074 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:40:48,075 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:40:48,267 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:40:49,075 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:40:51,268 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:40:53,966 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:40:53,966 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:40:53,966 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:40:53,967 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:40:54,269 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:40:54,967 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:40:57,270 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:40:59,859 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:40:59,860 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:40:59,860 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:40:59,860 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:41:00,271 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:41:00,861 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:41:01,272 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:41:02,676 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:41:02,677 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:41:05,766 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:41:05,766 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:41:05,766 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:41:05,767 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:41:06,273 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:41:06,767 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:41:07,273 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:41:11,660 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:41:11,661 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:41:11,661 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:41:11,662 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:41:12,275 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:41:12,662 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:41:13,275 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:41:14,158 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:41:17,561 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:41:17,562 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:41:17,562 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:41:17,562 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:41:17,676 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:41:17,677 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:41:17,797 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:41:18,276 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:41:19,277 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:41:22,798 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:41:23,463 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:41:23,464 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:41:23,464 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:41:23,465 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:41:24,278 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:41:25,278 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:41:28,466 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:41:29,360 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:41:29,361 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:41:29,361 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:41:29,362 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:41:30,279 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:41:31,280 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:41:32,676 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:41:32,677 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:41:33,896 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:41:35,263 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:41:35,264 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:41:35,264 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:41:35,264 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:41:35,281 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:41:37,281 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:41:39,265 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:41:41,170 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:41:41,171 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:41:41,171 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:41:41,171 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:41:41,282 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:41:43,283 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:41:44,154 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:41:45,155 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:41:47,078 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:41:47,078 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:41:47,079 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:41:47,080 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:41:47,284 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:41:47,677 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:41:47,677 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:41:49,285 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:41:50,866 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:41:52,978 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:41:52,979 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:41:52,979 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:41:52,979 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:41:53,286 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:41:55,287 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:41:55,980 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:41:58,876 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:41:58,877 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:41:58,877 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:41:58,877 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:41:59,288 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:42:01,288 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:42:01,878 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:42:02,677 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:42:02,677 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:42:04,774 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:42:04,775 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:42:04,775 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:42:04,775 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:42:05,289 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:42:07,290 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:42:07,776 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:42:10,674 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:42:10,675 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:42:10,675 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:42:10,675 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:42:11,291 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:42:13,292 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:42:13,676 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:42:14,155 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:42:16,585 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:42:16,585 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:42:16,586 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:42:16,586 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:42:17,293 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:42:17,677 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:42:17,677 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:42:18,873 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:42:19,294 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:42:22,477 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:42:22,478 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:42:22,478 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:42:22,479 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:42:23,295 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:42:24,479 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:42:25,295 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:42:28,399 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:42:28,400 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:42:28,400 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:42:28,401 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:42:29,297 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:42:30,401 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:42:31,297 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:42:32,677 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:42:32,677 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:42:34,315 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:42:34,316 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:42:34,316 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:42:34,316 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:42:35,298 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:42:36,317 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:42:37,299 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:42:40,205 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:42:40,206 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:42:40,206 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:42:40,206 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:42:40,300 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:42:42,207 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:42:43,301 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:42:44,155 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:42:46,099 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:42:46,099 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:42:46,100 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:42:46,100 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:42:46,302 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:42:47,677 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:42:47,677 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:42:47,849 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:42:49,303 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:42:52,003 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:42:52,004 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:42:52,004 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:42:52,004 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:42:52,304 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:42:53,005 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:42:53,304 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:42:57,896 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:42:57,897 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:42:57,897 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:42:57,897 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:42:58,306 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:42:58,898 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:42:59,306 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:43:02,677 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:43:02,677 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:43:03,798 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:43:03,798 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:43:03,798 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:43:03,799 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:43:04,307 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:43:04,799 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:43:05,308 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:43:09,699 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:43:09,699 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:43:09,700 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:43:09,700 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:43:10,309 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:43:10,700 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:43:11,310 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:43:14,156 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:43:15,605 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:43:15,606 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:43:15,606 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:43:15,607 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:43:16,311 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:43:16,607 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:43:17,311 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:43:17,677 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:43:17,677 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:43:21,526 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:43:21,526 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:43:21,527 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:43:21,528 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:43:22,313 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:43:22,528 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:43:23,313 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:43:27,433 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:43:27,434 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:43:27,434 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:43:27,434 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:43:28,315 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:43:28,434 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:43:29,315 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:43:32,677 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:43:32,678 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:43:33,338 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:43:33,339 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:43:33,339 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:43:33,339 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:43:34,316 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:43:34,340 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:43:35,317 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:43:39,238 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:43:39,239 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:43:39,239 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:43:39,239 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:43:39,318 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:43:40,240 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:43:41,319 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:43:44,156 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:43:45,164 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:43:45,164 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:43:45,165 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:43:45,165 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:43:45,320 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:43:46,165 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:43:47,321 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:43:47,677 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:43:47,678 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:43:51,057 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:43:51,058 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:43:51,058 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:43:51,058 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:43:51,322 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:43:52,059 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:43:53,322 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:43:56,977 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:43:56,978 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:43:56,978 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:43:56,978 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:43:57,324 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:43:57,979 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:43:59,324 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:44:02,677 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:44:02,678 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:44:02,879 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:44:02,879 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:44:02,880 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:44:02,880 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:44:03,325 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:44:03,880 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:44:05,326 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:44:08,798 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:44:08,799 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:44:08,799 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:44:08,799 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:44:09,327 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:44:09,799 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:44:11,328 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:44:14,157 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:44:14,700 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:44:14,700 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:44:14,701 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:44:14,701 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:44:15,329 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:44:15,701 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:44:17,330 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:44:17,678 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:44:17,678 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:44:20,605 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:44:20,610 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:44:20,610 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:44:20,610 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:44:21,331 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:44:21,611 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:44:23,331 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:44:26,516 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:44:26,516 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:44:26,517 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:44:26,517 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:44:27,333 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:44:27,517 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:44:29,333 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:44:32,436 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:44:32,437 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:44:32,437 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:44:32,437 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:44:32,678 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:44:32,678 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:44:32,899 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:44:33,335 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:44:35,335 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:44:37,900 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:44:38,335 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:44:38,336 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:44:38,336 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:44:38,337 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:44:38,337 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:44:41,338 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:44:43,338 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:44:44,157 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:44:44,243 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:44:44,243 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:44:44,243 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:44:44,244 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:44:44,339 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:44:47,340 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:44:47,678 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:44:47,678 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:44:48,871 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:44:50,143 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:44:50,144 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:44:50,144 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:44:50,144 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:44:50,341 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:44:51,341 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:44:54,145 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:44:56,045 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:44:56,045 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:44:56,046 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:44:56,047 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:44:56,342 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:44:57,343 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:45:00,047 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:45:01,959 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:45:01,960 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:45:01,960 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:45:01,960 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:45:02,344 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:45:02,678 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:45:02,678 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:45:03,344 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:45:05,871 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:45:07,862 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:45:07,862 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:45:07,863 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:45:07,863 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:45:08,345 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:45:09,346 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:45:11,864 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:45:13,773 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:45:13,774 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:45:13,774 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:45:13,775 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:45:14,158 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:45:14,347 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:45:15,347 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:45:17,159 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:45:17,678 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:45:17,678 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:45:19,674 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:45:19,674 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:45:19,674 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:45:19,675 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:45:20,349 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:45:21,349 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:45:22,675 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:45:25,576 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:45:25,577 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:45:25,577 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:45:25,577 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:45:26,350 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:45:27,351 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:45:28,578 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:45:31,486 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:45:31,487 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:45:31,487 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:45:31,487 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:45:32,352 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:45:32,678 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:45:32,678 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:45:33,352 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:45:33,866 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:45:37,382 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:45:37,383 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:45:37,383 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:45:37,383 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:45:38,354 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:45:39,354 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:45:39,384 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:45:43,306 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:45:43,307 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:45:43,307 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:45:43,307 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:45:43,355 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:45:44,159 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:45:45,159 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:45:45,356 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:45:47,678 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:45:47,679 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:45:49,194 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:45:49,194 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:45:49,195 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:45:49,195 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:45:49,357 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:45:50,195 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:45:51,357 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:45:55,085 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:45:55,086 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:45:55,086 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:45:55,086 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:45:55,358 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:45:56,087 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:45:57,359 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:46:00,985 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:46:00,986 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:46:00,986 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:46:00,986 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:46:01,360 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:46:01,987 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:46:02,678 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:46:02,679 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:46:03,360 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:46:06,886 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:46:06,887 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:46:06,887 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:46:06,887 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:46:07,361 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:46:07,888 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:46:09,362 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:46:12,803 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:46:12,804 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:46:12,804 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:46:12,804 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:46:13,363 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:46:13,805 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:46:14,159 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:46:15,363 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:46:17,679 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:46:17,679 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:46:18,694 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:46:18,695 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:46:18,695 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:46:18,695 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:46:19,364 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:46:19,696 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:46:21,365 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:46:24,593 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:46:24,593 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:46:24,594 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:46:24,594 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:46:25,366 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:46:25,594 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:46:27,366 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:46:30,495 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:46:30,496 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:46:30,496 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:46:30,497 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:46:31,368 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:46:31,497 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:46:32,679 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:46:32,679 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:46:33,368 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:46:36,391 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:46:36,391 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:46:36,392 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:46:36,392 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:46:37,369 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:46:37,392 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:46:39,370 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:46:42,287 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:46:42,287 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:46:42,287 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:46:42,288 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:46:42,370 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:46:43,288 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:46:43,371 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:46:44,160 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:46:47,679 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:46:47,679 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:46:48,179 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:46:48,180 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:46:48,180 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:46:48,180 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:46:48,372 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:46:49,181 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:46:49,372 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:46:54,074 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:46:54,075 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:46:54,075 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:46:54,076 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:46:54,374 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:46:55,076 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:46:55,374 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:46:59,969 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:46:59,970 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:46:59,970 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:46:59,971 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:47:00,375 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:47:00,971 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:47:01,375 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:47:02,679 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:47:02,679 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:47:05,864 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:47:05,865 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:47:05,865 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:47:05,865 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:47:06,377 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:47:06,866 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:47:07,377 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:47:11,763 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:47:11,763 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:47:11,764 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:47:11,764 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:47:12,378 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:47:12,764 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:47:13,378 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:47:14,160 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:47:17,686 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:47:17,686 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:47:17,686 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:47:17,687 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:47:17,687 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:47:17,687 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:47:17,879 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:47:18,380 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:47:19,380 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:47:22,880 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:47:23,575 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:47:23,576 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:47:23,576 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:47:23,577 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:47:24,381 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:47:25,382 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:47:28,577 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:47:29,507 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:47:29,508 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:47:29,508 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:47:29,508 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:47:30,383 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:47:31,383 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:47:32,686 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:47:32,686 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:47:33,914 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:47:35,398 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:47:35,402 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:47:35,403 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:47:35,403 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:47:36,384 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:47:37,385 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:47:39,404 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:47:41,323 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:47:41,324 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:47:41,324 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:47:41,324 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:47:41,386 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:47:43,386 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:47:44,161 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:47:45,161 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:47:47,237 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:47:47,238 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:47:47,238 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:47:47,238 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:47:47,387 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:47:47,686 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:47:47,686 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:47:49,388 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:47:50,844 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:47:53,139 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:47:53,140 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:47:53,140 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:47:53,140 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:47:53,389 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:47:55,389 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:47:56,141 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:47:59,032 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:47:59,033 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:47:59,033 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:47:59,033 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:47:59,390 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:48:01,391 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:48:02,034 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:48:02,686 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:48:02,687 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:48:04,945 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:48:04,945 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:48:04,946 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:48:04,946 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:48:05,392 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:48:07,392 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:48:07,946 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:48:10,855 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:48:10,856 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:48:10,856 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:48:10,857 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:48:11,393 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:48:13,394 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:48:13,857 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:48:14,161 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:48:16,747 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:48:16,748 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:48:16,748 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:48:16,748 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:48:17,395 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:48:17,686 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:48:17,687 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:48:18,905 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:48:19,396 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:48:23,906 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:48:28,907 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:48:32,686 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:48:32,687 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:48:33,399 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:48:34,846 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:48:37,367 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:48:37,368 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:48:37,368 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:48:37,400 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:48:37,509 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:48:39,401 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:48:40,510 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:48:43,226 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:48:43,227 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:48:43,228 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:48:43,247 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:48:43,402 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:48:44,162 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:48:45,402 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:48:46,162 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:48:47,687 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:48:47,687 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:48:49,076 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:48:49,077 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:48:49,077 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:48:49,077 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:48:49,403 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:48:51,404 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:48:52,078 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:48:54,947 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:48:54,948 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:48:54,948 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:48:54,949 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:48:55,404 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:48:57,405 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:48:57,949 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:49:00,817 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:49:00,817 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:49:00,817 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:49:00,818 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:49:01,406 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:49:02,687 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:49:02,687 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:49:03,407 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:49:03,878 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:49:06,689 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:49:06,690 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:49:06,690 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:49:06,690 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:49:07,407 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:49:09,408 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:49:09,691 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:49:14,162 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:49:15,163 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:49:17,891 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:49:17,892 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:49:21,107 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:49:22,274 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:49:22,275 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:49:22,275 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:49:22,275 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:49:22,411 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:49:23,411 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:49:26,276 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:49:28,084 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:49:28,085 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:49:28,085 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:49:28,085 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:49:28,413 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:49:29,413 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:49:32,086 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:49:32,891 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:49:32,891 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:49:33,970 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:49:33,970 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:49:33,970 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:49:33,971 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:49:34,414 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:49:35,414 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:49:37,972 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:49:39,848 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:49:39,849 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:49:39,849 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:49:39,849 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:49:40,415 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:49:41,416 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:49:43,850 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:49:44,163 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:49:45,727 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:49:45,728 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:49:45,728 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:49:45,729 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:49:46,417 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:49:47,417 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:49:47,891 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:49:47,892 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:49:49,121 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:49:51,601 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:49:51,601 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:49:51,602 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:49:51,602 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:49:52,418 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:49:53,419 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:49:54,603 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:49:57,499 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:49:57,500 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:49:57,500 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:49:57,500 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:49:58,420 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:49:59,420 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:50:00,501 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:50:02,891 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:50:02,892 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:50:03,434 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:50:03,434 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:50:03,435 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:50:03,435 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:50:04,421 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:50:05,421 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:50:06,436 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:50:09,268 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:50:09,268 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:50:09,268 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:50:09,269 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:50:09,422 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:50:11,423 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:50:12,269 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:50:14,163 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:50:15,162 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:50:15,163 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:50:15,163 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:50:15,163 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:50:15,424 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:50:17,424 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:50:17,891 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:50:17,891 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:50:17,892 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:50:21,126 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:50:21,127 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:50:21,127 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:50:21,127 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:50:21,425 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:50:23,127 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:50:23,425 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:50:26,983 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:50:26,984 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:50:26,984 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:50:26,984 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:50:27,426 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:50:28,985 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:50:29,427 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:50:32,882 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:50:32,883 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:50:32,883 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:50:32,883 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:50:32,893 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:50:32,893 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:50:33,428 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:50:34,058 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:50:35,428 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:50:38,826 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:50:38,827 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:50:38,827 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:50:38,827 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:50:39,430 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:50:39,828 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:50:41,431 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:50:44,164 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:50:44,679 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:50:44,679 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:50:44,679 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:50:44,680 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:50:45,432 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:50:45,680 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:50:47,432 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:50:47,892 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:50:47,893 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:50:50,630 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:50:50,631 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:50:50,631 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:50:50,631 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:50:51,433 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:50:51,434 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:50:51,632 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:50:56,489 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:50:56,492 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:50:56,492 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:50:56,492 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:50:57,435 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:50:57,435 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:50:57,493 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:51:02,386 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:51:02,386 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:51:02,387 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:51:02,387 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:51:02,436 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:51:02,892 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:51:02,893 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:51:02,893 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:51:03,437 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:51:08,126 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:51:08,318 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:51:08,319 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:51:08,319 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:51:08,320 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:51:08,438 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:51:09,438 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:51:13,321 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:51:14,164 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:51:14,165 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:51:14,165 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:51:14,166 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:51:14,166 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:51:14,440 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:51:15,440 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:51:17,892 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:51:17,893 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:51:19,115 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:51:20,122 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:51:20,122 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:51:20,123 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:51:20,123 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:51:20,441 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:51:21,442 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:51:24,124 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:51:25,979 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:51:25,979 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:51:25,980 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:51:25,980 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:51:26,443 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:51:27,443 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:51:29,981 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:51:31,883 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:51:31,884 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:51:31,884 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:51:31,884 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:51:32,445 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:51:32,893 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:51:32,893 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:51:33,445 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:51:35,015 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:51:37,826 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:51:37,827 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:51:37,827 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:51:37,827 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:51:38,446 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:51:39,447 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:51:40,828 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:51:43,670 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:51:43,671 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:51:43,671 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:51:43,671 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:51:44,165 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:51:44,448 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:51:45,448 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:51:46,165 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:51:47,893 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:51:47,893 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:51:49,606 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:51:49,606 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:51:49,607 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:51:49,607 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:51:50,450 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:51:51,450 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:51:51,607 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:51:55,470 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:51:55,471 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:51:55,471 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:51:55,471 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:51:56,451 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:51:57,452 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:51:57,472 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:52:01,358 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:52:01,358 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:52:01,359 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:52:01,359 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:52:01,453 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:52:02,893 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:52:02,893 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:52:02,893 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:52:03,453 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:52:07,298 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:52:07,299 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:52:07,299 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:52:07,299 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:52:07,454 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:52:08,299 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:52:09,455 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:52:13,145 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:52:13,146 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:52:13,146 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:52:13,146 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:52:13,456 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:52:13,456 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:52:14,147 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:52:14,165 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:52:17,893 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:52:17,893 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:52:19,070 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:52:19,070 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:52:19,071 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:52:19,071 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:52:19,458 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:52:19,458 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:52:20,071 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:52:24,930 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:52:24,930 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:52:24,930 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:52:24,931 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:52:25,460 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:52:25,460 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:52:25,931 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:52:30,832 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:52:30,833 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:52:30,833 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:52:30,833 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:52:31,461 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:52:31,461 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:52:31,834 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:52:32,893 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:52:32,893 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:52:36,766 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:52:36,767 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:52:36,767 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:52:36,767 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:52:37,463 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:52:37,463 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:52:37,767 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:52:42,615 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:52:42,615 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:52:42,616 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:52:42,617 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:52:43,465 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:52:43,465 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:52:43,617 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:52:44,166 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:52:47,893 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:52:47,894 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:52:48,558 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:52:48,559 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:52:48,559 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:52:48,559 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:52:49,466 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:52:49,466 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:52:49,559 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:52:54,393 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:52:54,393 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:52:54,393 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:52:54,394 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:52:54,467 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:52:55,394 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:52:55,468 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:53:00,299 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:53:00,300 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:53:00,300 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:53:00,300 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:53:00,469 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:53:01,301 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:53:01,469 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:53:02,893 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:53:02,894 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:53:06,246 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:53:06,246 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:53:06,247 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:53:06,247 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:53:06,470 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:53:07,247 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:53:07,471 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:53:12,092 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:53:12,093 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:53:12,093 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:53:12,093 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:53:12,472 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:53:13,094 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:53:13,472 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:53:14,166 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:53:17,893 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:53:17,894 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:53:18,030 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:53:18,061 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:53:18,061 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:53:18,061 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:53:18,474 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:53:19,062 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:53:19,474 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:53:23,934 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:53:23,934 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:53:23,935 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:53:23,935 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:53:24,475 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:53:24,935 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:53:25,475 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:53:29,786 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:53:29,786 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:53:29,786 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:53:29,787 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:53:30,477 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:53:30,787 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:53:31,477 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:53:32,893 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:53:32,894 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:53:35,734 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:53:35,734 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:53:35,735 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:53:35,735 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:53:36,478 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:53:36,735 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:53:37,478 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:53:41,592 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:53:41,593 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:53:41,593 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:53:41,593 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:53:42,480 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:53:42,593 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:53:43,480 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:53:44,167 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:53:47,485 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:53:47,486 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:53:47,486 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:53:47,486 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:53:47,894 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:53:47,894 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:53:47,894 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:53:48,481 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:53:49,481 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:53:53,056 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:53:53,430 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:53:53,430 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:53:53,431 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:53:53,431 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:53:53,482 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:53:55,483 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:53:58,432 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:53:59,301 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:53:59,302 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:53:59,302 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:53:59,302 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:53:59,484 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:53:59,484 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:54:02,894 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:54:02,894 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:54:04,089 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:54:05,238 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:54:05,242 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:54:05,242 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:54:05,242 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:54:05,485 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:54:05,485 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:54:09,243 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:54:11,087 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:54:11,088 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:54:11,088 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:54:11,089 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:54:11,486 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:54:11,487 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:54:14,167 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:54:15,168 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:54:16,972 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:54:16,972 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:54:16,972 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:54:16,973 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:54:17,488 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:54:17,488 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:54:17,894 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:54:17,894 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:54:21,080 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:54:22,914 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:54:22,914 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:54:22,915 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:54:22,915 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:54:23,489 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:54:23,490 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:54:26,916 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:54:28,767 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:54:28,768 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:54:28,768 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:54:28,768 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:54:29,491 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:54:29,491 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:54:32,769 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:54:32,894 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:54:32,894 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:54:34,702 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:54:34,702 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:54:34,703 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:54:34,703 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:54:35,492 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:54:35,493 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:54:38,704 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:54:40,560 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:54:40,560 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:54:40,560 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:54:40,561 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:54:41,494 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:54:41,494 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:54:44,168 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:54:44,169 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:54:46,461 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:54:46,462 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:54:46,462 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:54:46,462 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:54:46,495 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:54:47,495 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:54:47,894 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:54:47,894 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:54:50,092 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:54:52,402 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:54:52,403 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:54:52,403 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:54:52,403 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:54:52,497 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:54:53,497 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:54:55,404 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:54:58,268 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:54:58,269 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:54:58,269 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:54:58,270 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:54:58,498 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:54:59,498 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:55:01,270 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:55:02,894 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:55:02,895 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:55:04,210 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:55:04,210 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:55:04,211 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:55:04,211 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:55:04,500 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:55:05,500 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:55:07,212 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:55:10,077 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:55:10,078 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:55:10,078 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:55:10,078 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:55:10,501 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:55:11,502 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:55:13,079 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:55:14,168 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:55:15,969 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:55:15,969 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:55:15,970 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:55:15,970 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:55:16,503 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:55:17,504 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:55:17,894 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:55:17,895 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:55:19,034 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:55:21,910 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:55:21,910 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:55:21,911 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:55:21,911 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:55:22,505 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:55:23,506 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:55:24,912 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:55:27,767 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:55:27,767 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:55:27,767 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:55:27,768 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:55:28,507 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:55:29,507 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:55:30,768 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:55:32,895 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:55:32,895 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:55:33,660 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:55:33,660 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:55:33,660 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:55:33,661 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:55:34,509 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:55:35,509 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:55:36,661 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:55:39,574 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:55:39,574 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:55:39,574 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:55:39,575 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:55:40,511 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:55:41,511 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:55:42,575 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:55:44,169 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:55:45,472 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:55:45,473 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:55:45,473 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:55:45,473 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:55:45,512 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:55:47,513 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:55:47,894 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:55:47,895 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:55:47,895 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:55:51,414 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:55:51,414 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:55:51,415 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:55:51,415 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:55:51,514 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:55:51,514 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:55:53,415 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:55:57,275 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:55:57,275 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:55:57,275 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:55:57,276 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:55:57,516 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:55:57,516 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:55:59,276 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:56:02,895 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:56:02,895 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:56:03,214 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:56:03,214 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:56:03,215 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:56:03,215 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:56:03,518 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:56:03,518 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:56:05,215 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:56:09,070 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:56:09,071 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:56:09,071 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:56:09,071 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:56:09,520 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:56:09,520 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:56:11,072 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:56:14,169 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:56:14,987 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:56:14,988 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:56:14,988 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:56:14,988 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:56:15,522 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:56:15,522 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:56:16,989 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:56:17,895 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:56:17,895 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:56:20,930 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:56:20,930 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:56:20,931 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:56:20,931 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:56:21,523 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:56:21,523 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:56:22,931 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:56:26,789 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:56:26,789 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:56:26,790 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:56:26,790 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:56:27,525 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:56:27,525 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:56:28,790 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:56:32,687 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:56:32,688 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:56:32,688 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:56:32,688 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:56:32,895 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:56:32,895 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:56:33,526 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:56:33,526 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:56:34,119 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:56:38,622 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:56:38,622 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:56:38,623 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:56:38,623 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:56:39,528 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:56:39,623 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:56:40,528 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:56:44,170 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:56:44,470 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:56:44,471 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:56:44,471 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:56:44,471 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:56:44,529 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:56:45,472 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:56:46,530 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:56:47,896 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:56:47,896 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:56:50,369 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:56:50,369 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:56:50,369 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:56:50,370 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:56:50,531 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:56:51,370 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:56:52,531 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:56:56,274 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:56:56,275 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:56:56,275 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:56:56,275 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:56:56,532 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:56:57,276 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:56:58,533 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:57:02,188 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:57:02,188 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:57:02,189 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:57:02,189 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:57:02,534 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:57:02,895 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:57:02,895 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:57:02,896 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:57:04,535 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:57:08,085 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:57:08,130 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:57:08,130 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:57:08,131 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:57:08,131 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:57:08,536 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:57:10,536 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:57:13,132 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:57:13,986 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:57:13,986 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:57:13,987 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:57:13,987 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:57:14,174 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:57:14,537 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:57:16,538 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:57:17,895 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:57:17,896 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:57:19,083 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:57:19,918 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:57:19,918 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:57:19,919 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:57:19,919 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:57:20,539 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:57:22,540 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:57:24,920 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:57:25,762 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:57:25,763 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:57:25,763 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:57:25,763 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:57:26,541 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:57:28,542 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:57:30,764 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:57:31,666 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:57:31,667 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:57:31,667 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:57:31,667 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:57:32,543 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:57:32,895 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:57:32,896 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:57:34,543 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:57:36,125 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:57:37,568 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:57:37,569 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:57:37,569 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:57:37,569 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:57:38,544 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:57:38,544 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:57:41,570 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:57:43,466 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:57:43,467 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:57:43,467 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:57:43,468 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:57:43,546 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:57:44,171 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:57:44,546 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:57:47,172 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:57:47,896 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:57:47,896 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:57:49,418 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:57:49,418 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:57:49,419 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:57:49,420 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:57:49,547 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:57:50,547 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:57:52,420 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:57:55,330 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:57:55,330 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:57:55,330 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:57:55,331 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:57:55,549 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:57:56,549 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:57:58,331 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:58:01,228 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:58:01,228 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:58:01,228 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:58:01,229 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:58:01,550 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:58:02,551 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:58:02,896 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:58:02,896 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:58:04,035 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:58:07,158 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:58:07,158 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:58:07,159 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:58:07,159 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:58:07,552 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:58:08,552 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:58:09,159 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:58:13,019 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:58:13,020 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:58:13,020 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:58:13,021 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:58:13,553 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:58:14,171 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:58:14,172 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:58:14,554 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:58:17,896 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:58:17,896 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:58:18,938 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:58:18,938 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:58:18,939 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:58:18,939 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:58:19,555 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:58:19,939 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:58:20,556 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:58:24,793 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:58:24,793 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:58:24,793 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:58:24,794 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:58:25,557 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:58:25,794 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:58:26,557 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:58:30,702 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:58:30,703 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:58:30,703 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:58:30,703 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:58:31,559 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:58:31,704 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:58:32,559 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:58:32,897 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:58:32,898 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:58:37,130 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:58:42,131 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:58:44,172 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:58:46,562 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:58:47,897 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:58:47,898 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:58:47,898 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:58:50,866 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:58:50,867 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:58:50,867 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:58:50,979 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:58:51,564 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:58:52,564 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:58:52,979 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:58:56,675 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:58:56,676 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:58:56,676 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:58:56,867 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:58:57,565 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:58:58,566 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:58:58,867 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:59:02,535 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:59:02,535 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:59:02,536 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:59:02,536 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:59:02,567 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:59:02,897 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:59:02,898 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:59:04,067 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:59:04,567 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:59:08,442 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:59:08,442 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:59:08,443 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:59:08,443 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:59:08,568 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:59:09,443 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:59:10,569 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:59:14,172 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:59:14,291 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:59:14,291 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:59:14,291 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:59:14,292 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:59:14,570 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:59:15,292 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:59:16,570 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:59:17,898 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:59:17,898 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:59:20,210 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:59:20,210 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:59:20,211 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:59:20,211 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:59:20,571 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:59:21,211 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:59:22,572 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:59:26,051 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:59:26,052 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:59:26,052 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:59:26,052 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:59:26,573 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:59:27,053 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:59:28,574 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:59:31,933 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:59:31,933 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:59:31,934 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:59:31,934 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:59:32,575 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:59:32,898 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:59:32,899 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:59:32,899 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:59:34,575 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:59:37,858 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:59:37,858 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:59:37,859 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:59:37,859 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:59:38,577 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:59:38,859 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:59:40,577 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:59:43,729 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:59:43,730 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:59:43,730 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:59:43,730 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:59:44,173 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 20:59:44,173 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:59:44,578 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:59:46,579 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:59:47,898 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 20:59:47,898 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 20:59:49,630 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:59:49,630 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:59:49,631 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:59:49,631 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:59:49,631 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:59:50,580 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 20:59:50,580 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:59:54,632 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 20:59:55,517 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 20:59:55,518 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 20:59:55,518 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 20:59:55,519 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 20:59:55,581 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 20:59:56,582 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:00:00,520 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:00:01,404 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:00:01,405 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:00:01,405 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:00:01,405 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:00:01,583 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:00:02,583 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:00:02,898 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:00:02,898 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:00:06,042 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:00:07,292 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:00:07,292 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:00:07,293 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:00:07,293 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:00:07,585 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:00:08,585 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:00:11,293 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:00:13,188 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:00:13,188 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:00:13,188 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:00:13,189 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:00:13,587 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:00:14,173 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:00:14,587 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:00:17,174 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:00:17,898 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:00:17,898 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:00:19,089 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:00:19,090 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:00:19,090 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:00:19,090 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:00:19,588 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:00:20,589 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:00:23,091 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:00:24,973 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:00:24,976 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:00:24,977 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:00:24,977 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:00:25,590 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:00:26,590 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:00:28,978 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:00:30,878 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:00:30,879 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:00:30,879 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:00:30,879 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:00:31,592 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:00:32,592 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:00:32,898 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:00:32,898 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:00:34,117 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:00:36,768 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:00:36,768 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:00:36,768 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:00:36,769 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:00:37,593 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:00:38,594 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:00:39,769 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:00:42,669 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:00:42,669 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:00:42,670 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:00:42,670 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:00:43,595 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:00:44,174 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:00:44,595 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:00:45,174 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:00:47,899 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:00:47,899 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:00:48,559 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:00:48,559 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:00:48,559 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:00:48,560 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:00:48,597 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:00:50,560 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:00:50,597 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:00:54,446 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:00:54,447 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:00:54,447 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:00:54,447 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:00:54,598 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:00:56,448 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:00:56,599 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:01:00,332 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:01:00,333 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:01:00,333 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:01:00,333 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:01:00,600 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:01:02,334 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:01:02,600 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:01:02,898 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:01:02,899 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:01:06,223 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:01:06,224 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:01:06,224 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:01:06,224 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:01:06,601 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:01:08,225 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:01:08,602 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:01:12,112 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:01:12,113 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:01:12,113 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:01:12,113 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:01:12,603 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:01:14,114 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:01:14,174 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:01:14,604 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:01:17,898 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:01:17,899 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:01:18,015 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:01:18,034 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:01:18,034 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:01:18,034 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:01:18,605 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:01:20,035 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:01:20,605 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:01:23,942 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:01:23,943 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:01:23,943 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:01:23,943 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:01:24,606 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:01:25,943 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:01:26,607 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:01:30,944 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:01:35,945 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:01:37,877 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:01:37,877 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:01:41,011 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:01:42,100 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:01:42,101 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:01:42,101 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:01:42,101 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:01:42,611 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:01:44,175 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:01:44,612 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:01:46,175 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:01:47,935 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:01:47,935 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:01:47,936 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:01:47,936 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:01:48,613 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:01:50,614 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:01:51,937 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:01:52,876 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:01:52,877 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:01:53,801 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:01:53,802 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:01:53,802 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:01:53,802 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:01:54,615 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:01:54,615 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:01:57,803 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:01:59,677 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:01:59,678 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:01:59,678 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:01:59,679 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:02:00,616 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:02:00,616 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:02:03,680 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:02:05,555 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:02:05,556 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:02:05,556 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:02:05,556 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:02:05,617 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:02:06,618 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:02:07,877 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:02:07,877 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:02:09,004 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:02:11,462 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:02:11,462 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:02:11,463 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:02:11,463 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:02:11,619 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:02:12,619 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:02:14,175 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:02:14,175 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:02:17,299 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:02:17,300 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:02:17,300 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:02:17,300 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:02:17,621 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:02:18,621 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:02:19,300 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:02:22,877 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:02:22,877 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:02:23,226 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:02:23,226 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:02:23,227 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:02:23,227 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:02:23,622 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:02:24,623 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:02:25,227 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:02:29,067 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:02:29,068 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:02:29,068 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:02:29,068 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:02:29,624 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:02:30,624 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:02:31,069 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:02:34,955 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:02:34,955 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:02:34,955 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:02:34,956 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:02:35,625 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:02:36,626 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:02:36,956 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:02:37,877 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:02:37,877 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:02:40,898 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:02:40,898 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:02:40,899 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:02:40,899 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:02:41,627 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:02:42,627 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:02:42,899 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:02:44,176 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:02:46,745 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:02:46,746 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:02:46,746 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:02:46,746 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:02:47,628 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:02:48,628 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:02:48,746 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:02:52,650 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:02:52,651 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:02:52,651 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:02:52,651 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:02:52,877 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:02:52,877 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:02:53,630 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:02:54,088 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:02:54,630 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:02:58,544 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:02:58,544 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:02:58,544 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:02:58,545 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:02:58,631 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:02:59,545 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:03:00,631 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:03:04,438 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:03:04,439 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:03:04,439 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:03:04,439 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:03:04,632 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:03:05,440 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:03:06,633 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:03:07,877 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:03:07,877 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:03:10,330 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:03:10,331 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:03:10,331 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:03:10,331 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:03:10,633 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:03:11,332 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:03:12,634 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:03:14,176 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:03:16,236 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:03:16,236 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:03:16,237 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:03:16,237 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:03:16,635 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:03:17,237 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:03:18,635 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:03:22,134 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:03:22,135 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:03:22,135 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:03:22,135 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:03:22,636 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:03:22,877 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:03:22,877 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:03:23,106 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:03:24,637 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:03:28,029 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:03:28,029 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:03:28,030 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:03:28,030 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:03:28,638 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:03:29,030 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:03:30,638 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:03:33,919 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:03:33,920 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:03:33,920 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:03:33,920 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:03:34,639 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:03:34,640 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:03:34,921 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:03:37,877 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:03:37,878 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:03:39,862 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:03:39,862 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:03:39,863 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:03:39,863 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:03:40,641 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:03:40,641 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:03:40,863 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:03:44,180 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:03:45,720 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:03:45,721 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:03:45,721 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:03:45,721 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:03:46,643 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:03:46,643 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:03:46,722 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:03:51,622 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:03:51,623 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:03:51,623 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:03:51,623 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:03:51,644 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:03:52,624 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:03:52,644 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:03:52,877 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:03:52,878 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:03:57,505 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:03:57,506 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:03:57,506 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:03:57,506 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:03:57,646 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:03:58,507 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:03:58,646 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:04:03,397 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:04:03,397 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:04:03,398 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:04:03,399 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:04:03,647 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:04:04,399 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:04:04,648 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:04:07,878 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:04:07,878 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:04:09,338 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:04:09,338 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:04:09,339 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:04:09,339 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:04:09,649 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:04:10,339 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:04:10,649 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:04:14,177 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:04:15,192 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:04:15,193 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:04:15,193 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:04:15,193 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:04:15,650 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:04:16,194 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:04:16,651 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:04:21,082 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:04:21,082 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:04:21,083 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:04:21,083 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:04:21,652 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:04:22,083 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:04:22,652 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:04:22,878 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:04:22,878 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:04:27,010 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:04:27,010 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:04:27,011 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:04:27,011 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:04:27,654 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:04:28,011 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:04:28,654 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:04:32,855 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:04:32,856 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:04:32,856 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:04:32,856 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:04:33,655 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:04:33,856 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:04:34,656 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:04:37,878 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:04:37,878 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:04:38,798 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:04:38,798 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:04:38,799 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:04:38,799 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:04:39,657 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:04:39,799 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:04:40,657 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:04:44,178 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:04:44,646 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:04:44,646 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:04:44,647 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:04:44,647 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:04:44,658 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:04:45,647 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:04:46,659 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:04:50,537 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:04:50,537 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:04:50,537 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:04:50,537 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:04:50,660 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:04:51,538 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:04:52,660 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:04:52,878 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:04:52,879 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:04:56,478 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:04:56,478 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:04:56,479 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:04:56,479 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:04:56,661 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:04:57,479 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:04:58,662 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:05:02,337 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:05:02,338 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:05:02,338 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:05:02,338 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:05:02,663 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:05:03,339 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:05:04,663 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:05:07,878 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:05:07,879 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:05:08,274 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:05:08,274 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:05:08,275 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:05:08,275 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:05:08,664 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:05:09,275 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:05:10,665 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:05:14,133 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:05:14,133 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:05:14,133 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:05:14,134 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:05:14,178 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:05:14,666 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:05:15,178 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:05:16,667 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:05:20,016 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:05:20,017 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:05:20,017 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:05:20,017 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:05:20,668 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:05:20,668 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:05:21,018 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:05:22,878 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:05:22,878 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:05:25,958 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:05:25,958 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:05:25,959 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:05:25,959 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:05:26,669 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:05:26,669 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:05:26,959 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:05:31,807 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:05:31,807 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:05:31,808 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:05:31,808 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:05:32,671 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:05:32,671 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:05:32,808 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:05:37,705 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:05:37,706 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:05:37,706 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:05:37,707 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:05:37,878 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:05:37,878 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:05:37,879 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:05:38,672 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:05:38,672 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:05:43,063 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:05:43,646 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:05:43,646 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:05:43,647 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:05:43,647 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:05:43,673 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:05:44,179 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:05:44,674 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:05:48,180 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:05:49,503 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:05:49,503 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:05:49,503 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:05:49,504 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:05:49,675 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:05:50,675 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:05:52,878 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:05:52,879 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:05:54,110 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:05:55,442 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:05:55,442 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:05:55,443 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:05:55,443 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:05:55,677 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:05:56,677 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:05:59,444 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:06:01,289 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:06:01,289 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:06:01,290 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:06:01,290 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:06:01,678 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:06:02,678 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:06:05,291 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:06:07,192 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:06:07,192 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:06:07,193 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:06:07,193 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:06:07,679 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:06:07,878 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:06:07,879 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:06:08,680 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:06:11,059 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:06:13,126 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:06:13,126 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:06:13,127 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:06:13,127 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:06:13,681 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:06:14,179 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:06:14,681 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:06:16,180 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:06:18,980 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:06:18,980 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:06:18,980 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:06:18,981 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:06:19,683 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:06:20,683 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:06:21,981 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:06:22,878 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:06:22,879 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:06:24,914 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:06:24,914 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:06:24,915 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:06:24,915 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:06:25,684 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:06:26,684 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:06:27,916 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:06:30,781 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:06:30,781 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:06:30,781 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:06:30,782 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:06:31,686 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:06:32,686 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:06:33,782 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:06:36,665 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:06:36,666 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:06:36,666 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:06:36,667 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:06:36,687 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:06:37,879 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:06:37,879 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:06:38,688 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:06:39,107 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:06:42,598 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:06:42,598 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:06:42,599 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:06:42,599 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:06:42,689 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:06:44,180 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:06:44,180 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:06:44,689 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:06:48,450 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:06:48,454 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:06:48,454 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:06:48,455 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:06:48,690 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:06:49,455 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:06:50,691 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:06:52,879 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:06:52,879 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:06:54,394 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:06:54,394 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:06:54,395 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:06:54,395 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:06:54,692 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:06:55,395 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:06:56,692 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:07:00,247 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:07:00,248 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:07:00,248 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:07:00,248 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:07:00,693 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:07:01,249 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:07:02,694 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:07:06,143 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:07:06,143 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:07:06,144 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:07:06,144 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:07:06,695 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:07:06,695 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:07:07,144 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:07:07,879 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:07:07,879 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:07:12,082 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:07:12,082 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:07:12,083 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:07:12,084 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:07:12,696 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:07:12,696 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:07:13,084 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:07:14,180 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:07:17,939 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:07:17,939 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:07:17,939 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:07:17,940 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:07:18,698 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:07:18,698 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:07:18,940 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:07:22,879 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:07:22,879 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:07:23,862 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:07:23,862 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:07:23,863 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:07:23,863 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:07:24,700 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:07:24,700 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:07:24,863 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:07:29,706 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:07:29,707 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:07:29,707 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:07:29,707 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:07:30,701 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:07:30,702 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:07:30,708 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:07:35,595 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:07:35,595 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:07:35,596 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:07:35,596 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:07:35,703 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:07:36,596 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:07:36,703 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:07:37,879 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:07:37,880 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:07:41,534 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:07:41,534 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:07:41,535 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:07:41,535 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:07:41,704 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:07:42,535 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:07:42,705 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:07:44,181 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:07:47,385 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:07:47,386 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:07:47,386 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:07:47,386 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:07:47,706 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:07:48,387 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:07:48,706 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:07:52,879 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:07:52,879 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:07:53,306 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:07:53,306 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:07:53,307 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:07:53,307 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:07:53,708 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:07:54,307 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:07:54,708 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:07:59,160 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:07:59,160 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:07:59,160 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:07:59,161 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:07:59,710 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:08:00,161 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:08:00,710 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:08:05,052 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:08:05,052 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:08:05,052 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:08:05,053 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:08:05,711 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:08:06,053 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:08:06,711 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:08:07,879 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:08:07,880 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:08:10,986 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:08:10,986 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:08:10,987 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:08:10,987 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:08:11,713 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:08:11,987 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:08:12,713 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:08:14,181 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:08:16,859 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:08:16,860 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:08:16,860 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:08:16,860 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:08:17,714 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:08:17,860 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:08:18,714 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:08:22,748 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:08:22,749 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:08:22,749 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:08:22,749 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:08:22,879 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:08:22,880 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:08:22,880 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:08:23,716 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:08:24,716 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:08:28,043 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:08:28,694 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:08:28,694 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:08:28,695 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:08:28,695 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:08:28,717 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:08:30,717 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:08:33,696 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:08:34,550 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:08:34,550 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:08:34,550 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:08:34,551 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:08:34,718 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:08:36,719 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:08:37,879 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:08:37,880 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:08:39,060 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:08:40,506 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:08:40,506 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:08:40,507 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:08:40,507 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:08:40,720 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:08:42,721 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:08:44,182 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:08:44,182 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:08:46,359 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:08:46,360 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:08:46,360 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:08:46,360 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:08:46,722 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:08:46,722 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:08:49,361 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:08:52,880 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:08:52,880 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:08:55,125 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:09:00,126 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:09:02,725 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:09:06,087 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:09:06,859 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:09:06,860 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:09:06,860 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:09:06,872 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:09:07,727 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:09:07,880 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:09:07,880 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:09:08,727 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:09:11,090 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:09:12,742 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:09:12,743 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:09:12,743 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:09:12,917 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:09:13,728 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:09:14,182 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:09:14,729 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:09:16,183 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:09:18,560 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:09:18,560 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:09:18,561 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:09:18,561 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:09:18,730 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:09:20,730 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:09:21,561 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:09:22,880 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:09:22,880 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:09:24,502 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:09:24,502 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:09:24,503 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:09:24,503 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:09:24,731 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:09:26,732 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:09:27,503 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:09:30,332 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:09:30,333 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:09:30,333 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:09:30,333 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:09:30,733 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:09:30,733 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:09:33,334 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:09:36,234 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:09:36,235 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:09:36,235 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:09:36,235 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:09:36,734 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:09:36,735 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:09:37,880 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:09:37,881 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:09:39,034 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:09:42,142 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:09:42,142 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:09:42,143 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:09:42,143 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:09:42,736 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:09:42,736 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:09:44,143 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:09:44,182 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:09:48,001 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:09:48,001 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:09:48,001 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:09:48,002 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:09:48,737 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:09:48,738 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:09:50,002 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:09:52,881 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:09:52,881 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:09:53,901 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:09:53,901 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:09:53,901 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:09:53,902 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:09:54,739 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:09:54,739 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:09:55,902 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:09:59,788 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:09:59,789 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:09:59,789 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:09:59,789 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:10:00,740 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:10:00,740 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:10:01,790 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:10:05,680 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:10:05,681 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:10:05,681 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:10:05,682 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:10:05,742 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:10:06,742 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:10:07,682 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:10:07,880 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:10:07,881 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:10:11,582 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:10:11,589 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:10:11,590 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:10:11,590 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:10:11,743 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:10:12,743 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:10:13,590 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:10:14,183 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:10:17,502 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:10:17,502 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:10:17,502 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:10:17,503 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:10:17,744 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:10:18,745 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:10:19,503 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:10:22,881 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:10:22,881 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:10:23,424 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:10:23,424 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:10:23,425 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:10:23,425 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:10:23,746 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:10:24,746 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:10:25,425 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:10:29,337 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:10:29,338 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:10:29,338 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:10:29,338 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:10:29,747 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:10:30,748 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:10:31,339 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:10:35,228 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:10:35,228 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:10:35,228 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:10:35,229 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:10:35,749 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:10:36,749 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:10:37,229 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:10:37,881 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:10:37,881 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:10:41,115 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:10:41,116 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:10:41,116 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:10:41,116 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:10:41,750 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:10:42,751 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:10:43,117 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:10:44,183 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:10:47,009 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:10:47,009 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:10:47,010 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:10:47,010 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:10:47,752 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:10:48,752 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:10:49,010 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:10:52,881 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:10:52,881 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:10:52,903 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:10:53,043 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:10:53,044 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:10:53,044 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:10:53,753 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:10:54,044 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:10:54,754 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:10:58,798 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:10:58,798 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:10:58,798 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:10:58,799 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:10:59,755 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:10:59,799 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:11:00,755 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:11:04,684 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:11:04,685 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:11:04,685 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:11:04,685 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:11:04,756 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:11:05,686 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:11:06,757 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:11:07,881 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:11:07,881 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:11:10,562 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:11:10,563 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:11:10,563 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:11:10,563 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:11:10,758 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:11:11,564 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:11:12,758 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:11:14,184 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:11:16,454 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:11:16,455 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:11:16,455 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:11:16,455 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:11:16,759 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:11:16,759 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:11:17,455 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:11:22,338 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:11:22,339 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:11:22,339 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:11:22,339 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:11:22,761 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:11:22,761 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:11:22,881 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:11:22,881 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:11:23,101 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:11:28,102 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:11:28,229 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:11:28,230 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:11:28,230 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:11:28,230 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:11:28,762 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:11:28,762 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:11:33,231 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:11:34,109 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:11:34,110 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:11:34,110 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:11:34,110 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:11:34,763 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:11:34,763 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:11:37,881 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:11:37,881 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:11:39,092 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:11:39,994 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:11:39,995 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:11:39,995 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:11:39,995 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:11:40,765 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:11:40,765 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:11:44,185 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:11:44,185 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:11:45,876 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:11:45,876 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:11:45,876 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:11:45,876 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:11:46,766 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:11:46,766 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:11:49,877 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:11:51,767 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:11:51,768 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:11:51,768 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:11:51,768 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:11:52,768 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:11:52,768 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:11:52,881 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:11:52,882 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:11:55,029 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:11:57,661 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:11:57,662 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:11:57,662 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:11:57,662 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:11:57,769 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:11:58,770 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:12:00,663 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:12:03,553 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:12:03,554 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:12:03,554 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:12:03,554 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:12:03,771 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:12:04,771 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:12:06,555 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:12:07,881 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:12:07,882 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:12:09,448 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:12:09,449 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:12:09,449 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:12:09,449 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:12:09,773 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:12:10,773 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:12:12,450 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:12:14,185 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:12:15,343 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:12:15,344 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:12:15,344 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:12:15,344 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:12:15,774 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:12:16,775 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:12:18,345 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:12:21,224 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:12:21,225 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:12:21,225 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:12:21,225 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:12:21,776 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:12:22,776 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:12:22,882 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:12:22,882 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:12:24,079 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:12:27,111 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:12:27,111 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:12:27,111 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:12:27,112 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:12:27,778 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:12:28,778 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:12:29,112 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:12:33,004 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:12:33,005 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:12:33,005 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:12:33,005 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:12:33,779 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:12:34,779 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:12:35,006 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:12:37,882 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:12:37,882 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:12:38,930 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:12:38,931 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:12:38,931 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:12:38,931 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:12:39,781 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:12:40,781 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:12:40,932 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:12:44,185 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:12:44,765 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:12:44,766 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:12:44,766 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:12:44,766 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:12:44,782 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:12:46,767 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:12:46,782 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:12:50,670 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:12:50,671 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:12:50,671 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:12:50,671 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:12:50,783 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:12:52,672 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:12:52,784 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:12:52,882 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:12:52,882 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:12:56,553 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:12:56,553 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:12:56,554 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:12:56,554 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:12:56,785 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:12:56,785 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:12:58,554 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:13:02,437 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:13:02,437 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:13:02,438 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:13:02,438 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:13:02,787 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:13:02,787 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:13:04,438 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:13:07,882 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:13:07,882 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:13:08,332 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:13:08,332 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:13:08,333 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:13:08,333 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:13:08,788 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:13:08,788 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:13:10,333 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:13:14,189 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:13:14,223 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:13:14,224 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:13:14,224 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:13:14,224 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:13:14,790 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:13:14,790 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:13:16,225 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:13:20,111 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:13:20,112 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:13:20,112 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:13:20,112 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:13:20,791 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:13:20,791 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:13:22,113 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:13:22,882 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:13:22,882 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:13:26,006 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:13:26,006 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:13:26,007 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:13:26,007 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:13:26,793 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:13:26,793 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:13:28,007 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:13:31,891 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:13:31,891 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:13:31,892 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:13:31,892 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:13:32,794 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:13:32,794 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:13:33,893 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:13:37,784 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:13:37,785 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:13:37,785 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:13:37,785 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:13:37,796 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:13:37,882 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:13:37,883 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:13:38,796 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:13:39,112 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:13:43,710 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:13:43,711 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:13:43,711 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:13:43,711 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:13:43,797 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:13:44,186 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:13:44,187 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:13:45,798 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:13:49,188 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:13:49,552 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:13:49,553 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:13:49,553 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:13:49,553 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:13:49,799 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:13:51,799 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:13:54,554 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:13:59,555 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:14:03,096 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:14:03,096 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:14:05,274 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:14:07,842 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:14:07,842 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:14:07,843 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:14:07,843 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:14:08,803 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:14:09,804 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:14:10,844 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:14:13,688 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:14:13,689 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:14:13,689 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:14:13,689 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:14:13,805 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:14:14,187 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:14:15,805 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:14:16,188 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:14:18,096 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:14:18,096 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:14:19,586 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:14:19,586 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:14:19,587 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:14:19,587 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:14:19,806 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:14:21,588 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:14:21,807 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:14:25,409 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:14:25,410 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:14:25,410 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:14:25,410 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:14:25,807 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:14:27,411 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:14:27,808 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:14:31,310 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:14:31,311 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:14:31,311 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:14:31,311 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:14:31,809 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:14:33,096 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:14:33,096 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:14:33,097 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:14:33,809 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:14:37,230 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:14:37,230 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:14:37,231 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:14:37,231 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:14:37,811 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:14:38,231 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:14:39,811 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:14:43,069 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:14:43,070 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:14:43,070 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:14:43,070 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:14:43,812 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:14:44,071 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:14:44,187 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:14:45,813 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:14:48,096 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:14:48,097 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:14:48,990 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:14:48,991 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:14:48,991 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:14:48,991 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:14:49,814 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:14:49,991 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:14:51,814 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:14:54,828 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:14:54,829 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:14:54,829 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:14:54,829 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:14:55,815 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:14:55,829 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:14:57,816 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:15:00,710 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:15:00,711 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:15:00,711 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:15:00,711 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:15:00,817 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:15:01,712 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:15:01,817 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:15:03,097 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:15:03,097 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:15:06,670 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:15:06,670 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:15:06,671 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:15:06,671 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:15:06,818 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:15:07,671 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:15:07,818 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:15:12,527 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:15:12,528 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:15:12,528 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:15:12,528 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:15:12,819 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:15:13,529 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:15:13,820 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:15:14,188 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:15:18,096 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:15:18,097 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:15:18,454 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:15:18,455 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:15:18,455 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:15:18,456 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:15:18,821 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:15:19,456 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:15:19,821 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:15:24,350 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:15:24,351 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:15:24,351 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:15:24,351 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:15:24,822 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:15:25,352 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:15:25,823 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:15:30,252 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:15:30,253 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:15:30,253 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:15:30,253 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:15:30,824 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:15:31,254 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:15:31,824 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:15:33,096 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:15:33,097 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:15:36,186 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:15:36,186 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:15:36,187 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:15:36,187 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:15:36,827 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:15:37,187 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:15:37,827 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:15:42,029 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:15:42,030 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:15:42,030 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:15:42,030 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:15:42,828 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:15:43,031 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:15:43,829 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:15:44,188 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:15:47,923 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:15:47,924 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:15:47,924 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:15:47,924 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:15:48,097 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:15:48,097 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:15:48,097 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:15:48,830 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:15:49,830 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:15:53,327 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:15:53,850 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:15:53,850 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:15:53,851 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:15:53,851 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:15:54,831 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:15:55,831 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:15:58,852 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:15:59,695 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:15:59,696 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:15:59,696 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:15:59,696 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:15:59,832 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:16:01,833 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:16:03,097 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:16:03,097 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:16:04,303 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:16:05,618 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:16:05,618 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:16:05,619 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:16:05,619 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:16:05,834 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:16:07,834 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:16:09,620 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:16:11,451 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:16:11,451 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:16:11,452 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:16:11,452 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:16:11,835 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:16:13,836 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:16:14,189 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:16:15,190 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:16:17,343 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:16:17,344 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:16:17,344 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:16:17,344 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:16:17,837 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:16:18,097 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:16:18,097 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:16:19,837 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:16:20,261 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:16:23,270 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:16:23,271 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:16:23,271 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:16:23,271 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:16:23,838 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:16:25,271 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:16:25,839 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:16:29,115 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:16:29,116 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:16:29,116 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:16:29,116 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:16:29,840 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:16:31,117 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:16:31,840 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:16:33,097 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:16:33,097 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:16:35,050 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:16:35,055 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:16:35,055 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:16:35,055 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:16:35,841 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:16:37,056 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:16:37,842 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:16:40,887 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:16:40,888 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:16:40,888 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:16:40,888 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:16:41,843 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:16:41,843 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:16:42,889 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:16:44,189 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:16:46,799 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:16:46,799 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:16:46,799 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:16:46,800 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:16:46,844 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:16:47,844 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:16:48,097 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:16:48,097 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:16:48,097 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:16:52,726 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:16:52,727 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:16:52,727 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:16:52,728 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:16:52,846 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:16:53,728 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:16:53,846 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:16:58,563 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:16:58,563 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:16:58,564 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:16:58,564 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:16:58,847 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:16:59,564 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:16:59,847 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:17:03,097 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:17:03,098 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:17:04,502 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:17:04,502 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:17:04,503 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:17:04,503 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:17:04,849 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:17:05,503 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:17:05,849 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:17:10,352 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:17:10,353 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:17:10,353 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:17:10,353 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:17:10,850 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:17:11,354 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:17:11,851 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:17:14,190 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:17:16,241 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:17:16,242 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:17:16,242 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:17:16,243 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:17:16,852 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:17:17,243 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:17:17,852 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:17:18,097 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:17:18,098 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:17:22,190 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:17:22,191 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:17:22,191 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:17:22,191 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:17:22,853 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:17:23,191 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:17:23,854 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:17:28,043 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:17:28,044 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:17:28,044 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:17:28,044 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:17:28,855 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:17:29,045 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:17:29,855 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:17:33,097 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:17:33,098 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:17:33,974 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:17:33,974 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:17:33,975 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:17:33,975 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:17:34,856 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:17:34,975 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:17:35,857 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:17:39,838 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:17:39,839 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:17:39,839 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:17:39,839 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:17:39,858 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:17:40,839 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:17:41,858 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:17:44,191 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:17:45,720 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:17:45,720 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:17:45,720 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:17:45,721 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:17:45,859 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:17:46,721 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:17:47,860 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:17:48,098 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:17:48,098 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:17:51,646 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:17:51,646 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:17:51,647 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:17:51,647 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:17:51,861 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:17:52,647 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:17:53,861 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:17:57,487 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:17:57,487 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:17:57,487 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:17:57,488 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:17:57,862 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:17:58,488 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:17:59,863 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:18:03,098 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:18:03,098 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:18:03,422 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:18:03,422 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:18:03,423 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:18:03,423 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:18:03,864 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:18:04,423 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:18:05,864 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:18:09,259 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:18:09,260 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:18:09,260 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:18:09,260 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:18:09,865 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:18:10,261 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:18:11,866 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:18:14,191 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:18:15,136 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:18:15,136 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:18:15,136 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:18:15,137 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:18:15,867 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:18:16,137 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:18:17,868 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:18:18,098 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:18:18,098 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:18:21,062 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:18:21,062 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:18:21,063 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:18:21,063 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:18:21,869 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:18:21,869 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:18:22,063 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:18:26,915 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:18:26,916 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:18:26,916 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:18:26,917 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:18:27,870 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:18:27,870 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:18:27,918 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:18:32,794 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:18:32,794 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:18:32,795 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:18:32,795 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:18:32,872 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:18:33,098 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:18:33,098 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:18:33,098 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:18:33,872 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:18:38,256 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:18:38,734 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:18:38,734 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:18:38,735 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:18:38,735 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:18:38,873 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:18:39,873 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:18:43,736 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:18:44,192 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:18:44,576 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:18:44,576 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:18:44,577 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:18:44,577 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:18:44,875 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:18:45,875 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:18:48,098 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:18:48,098 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:18:49,256 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:18:50,534 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:18:50,534 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:18:50,535 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:18:50,535 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:18:50,876 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:18:51,877 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:18:54,535 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:18:56,396 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:18:56,396 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:18:56,397 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:18:56,397 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:18:56,878 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:18:57,878 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:19:00,398 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:19:02,290 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:19:02,291 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:19:02,291 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:19:02,291 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:19:02,880 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:19:03,125 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:19:03,126 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:19:03,880 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:19:06,345 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:19:11,346 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:19:14,192 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:19:16,950 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:19:17,883 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:19:18,125 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:19:18,126 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:19:19,884 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:19:22,300 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:19:23,590 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:19:23,590 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:19:23,591 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:19:23,767 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:19:23,885 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:19:25,886 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:19:27,768 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:19:29,410 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:19:29,411 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:19:29,411 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:19:29,485 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:19:29,887 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:19:31,887 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:19:33,126 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:19:33,126 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:19:33,126 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:19:35,302 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:19:35,303 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:19:35,303 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:19:35,303 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:19:35,888 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:19:37,889 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:19:38,303 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:19:41,121 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:19:41,122 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:19:41,122 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:19:41,122 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:19:41,890 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:19:43,890 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:19:44,123 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:19:44,198 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:19:47,010 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:19:47,010 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:19:47,010 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:19:47,011 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:19:47,892 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:19:47,892 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:19:48,126 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:19:48,126 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:19:49,318 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:19:52,934 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:19:52,934 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:19:52,935 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:19:52,935 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:19:53,893 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:19:53,893 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:19:54,935 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:19:58,780 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:19:58,780 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:19:58,780 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:19:58,781 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:19:58,894 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:19:59,894 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:20:00,781 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:20:03,126 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:20:03,126 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:20:04,698 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:20:04,699 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:20:04,699 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:20:04,700 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:20:04,896 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:20:05,896 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:20:06,700 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:20:10,537 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:20:10,537 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:20:10,537 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:20:10,538 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:20:10,897 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:20:11,898 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:20:12,538 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:20:14,193 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:20:16,417 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:20:16,418 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:20:16,418 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:20:16,419 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:20:16,899 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:20:17,899 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:20:18,126 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:20:18,126 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:20:18,126 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:20:22,350 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:20:22,351 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:20:22,351 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:20:22,351 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:20:22,900 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:20:23,351 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:20:23,901 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:20:28,193 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:20:28,194 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:20:28,194 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:20:28,194 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:20:28,902 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:20:29,195 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:20:29,902 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:20:33,126 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:20:33,126 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:20:34,138 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:20:34,139 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:20:34,139 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:20:34,139 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:20:34,903 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:20:35,139 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:20:35,904 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:20:39,987 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:20:39,988 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:20:39,988 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:20:39,988 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:20:40,905 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:20:40,988 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:20:41,905 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:20:44,194 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:20:45,883 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:20:45,883 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:20:45,884 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:20:45,884 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:20:45,906 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:20:46,884 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:20:47,906 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:20:48,126 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:20:48,127 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:20:51,814 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:20:51,815 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:20:51,815 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:20:51,815 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:20:51,907 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:20:52,815 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:20:53,908 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:20:57,680 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:20:57,681 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:20:57,681 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:20:57,681 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:20:57,909 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:20:58,681 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:20:59,910 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:21:03,126 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:21:03,127 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:21:03,610 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:21:03,611 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:21:03,611 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:21:03,611 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:21:03,911 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:21:04,611 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:21:05,911 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:21:09,468 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:21:09,468 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:21:09,468 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:21:09,469 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:21:09,912 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:21:10,469 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:21:11,913 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:21:14,194 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:21:15,353 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:21:15,354 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:21:15,354 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:21:15,354 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:21:15,914 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:21:16,354 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:21:17,915 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:21:18,127 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:21:18,127 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:21:21,282 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:21:21,283 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:21:21,283 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:21:21,283 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:21:21,916 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:21:22,283 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:21:23,916 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:21:27,138 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:21:27,138 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:21:27,138 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:21:27,139 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:21:27,918 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:21:27,918 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:21:28,139 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:21:33,045 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:21:33,046 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:21:33,046 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:21:33,046 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:21:33,127 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:21:33,127 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:21:33,311 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:21:33,919 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:21:33,919 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:21:38,311 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:21:38,933 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:21:38,934 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:21:38,934 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:21:38,935 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:21:39,921 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:21:39,921 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:21:43,936 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:21:44,195 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:21:44,815 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:21:44,815 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:21:44,815 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:21:44,816 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:21:44,922 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:21:45,922 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:21:48,127 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:21:48,127 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:21:49,302 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:21:50,754 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:21:50,755 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:21:50,755 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:21:50,755 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:21:50,924 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:21:51,924 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:21:54,756 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:21:56,626 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:21:56,627 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:21:56,627 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:21:56,627 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:21:56,925 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:21:57,926 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:22:00,628 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:22:02,510 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:22:02,511 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:22:02,511 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:22:02,511 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:22:02,927 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:22:03,127 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:22:03,127 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:22:03,927 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:22:06,342 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:22:08,446 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:22:08,447 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:22:08,447 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:22:08,447 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:22:08,928 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:22:09,928 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:22:11,447 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:22:14,195 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:22:14,292 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:22:14,293 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:22:14,293 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:22:14,294 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:22:14,930 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:22:15,930 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:22:17,294 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:22:18,127 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:22:18,127 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:22:20,230 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:22:20,231 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:22:20,231 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:22:20,231 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:22:20,931 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:22:21,932 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:22:23,232 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:22:26,083 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:22:26,084 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:22:26,084 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:22:26,084 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:22:26,933 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:22:27,933 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:22:29,085 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:22:31,973 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:22:31,974 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:22:31,974 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:22:31,974 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:22:32,935 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:22:33,127 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:22:33,127 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:22:33,935 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:22:34,293 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:22:37,914 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:22:37,915 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:22:37,915 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:22:37,915 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:22:37,936 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:22:39,915 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:22:39,936 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:22:43,783 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:22:43,784 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:22:43,784 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:22:43,784 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:22:43,937 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:22:44,196 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:22:45,196 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:22:45,938 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:22:48,127 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:22:48,127 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:22:49,714 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:22:49,715 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:22:49,715 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:22:49,715 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:22:49,939 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:22:50,715 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:22:51,940 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:22:55,572 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:22:55,576 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:22:55,576 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:22:55,576 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:22:55,941 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:22:56,577 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:22:57,941 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:23:01,464 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:23:01,465 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:23:01,465 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:23:01,465 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:23:01,943 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:23:02,466 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:23:03,127 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:23:03,128 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:23:03,943 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:23:07,398 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:23:07,399 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:23:07,399 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:23:07,399 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:23:07,944 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:23:07,944 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:23:08,399 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:23:13,233 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:23:13,233 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:23:13,233 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:23:13,234 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:23:13,946 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:23:13,946 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:23:14,196 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:23:14,196 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:23:18,127 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:23:18,128 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:23:19,124 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:23:19,124 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:23:19,124 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:23:19,125 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:23:19,948 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:23:19,948 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:23:20,125 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:23:25,014 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:23:25,014 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:23:25,014 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:23:25,015 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:23:25,949 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:23:25,949 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:23:26,015 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:23:30,909 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:23:30,909 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:23:30,909 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:23:30,910 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:23:30,951 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:23:31,910 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:23:31,951 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:23:33,128 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:23:33,128 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:23:36,838 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:23:36,839 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:23:36,839 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:23:36,839 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:23:36,952 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:23:37,839 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:23:37,952 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:23:42,713 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:23:42,714 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:23:42,714 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:23:42,714 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:23:42,954 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:23:43,714 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:23:43,954 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:23:44,197 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:23:48,128 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:23:48,128 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:23:48,658 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:23:48,659 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:23:48,659 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:23:48,659 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:23:48,956 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:23:49,659 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:23:49,956 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:23:54,520 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:23:54,520 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:23:54,521 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:23:54,521 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:23:54,957 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:23:55,521 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:23:55,957 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:24:00,405 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:24:00,406 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:24:00,406 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:24:00,406 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:24:00,959 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:24:01,407 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:24:01,959 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:24:03,128 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:24:03,128 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:24:06,283 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:24:06,284 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:24:06,284 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:24:06,284 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:24:06,960 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:24:07,284 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:24:07,961 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:24:12,167 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:24:12,167 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:24:12,168 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:24:12,168 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:24:12,962 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:24:13,168 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:24:13,962 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:24:14,197 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:24:18,066 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:24:18,067 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:24:18,067 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:24:18,067 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:24:18,128 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:24:18,128 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:24:18,334 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:24:18,963 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:24:19,964 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:24:23,335 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:24:23,982 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:24:23,983 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:24:23,983 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:24:23,983 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:24:24,965 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:24:25,965 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:24:28,984 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:24:29,822 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:24:29,822 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:24:29,822 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:24:29,823 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:24:29,966 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:24:31,967 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:24:33,129 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:24:33,129 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:24:34,349 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:24:35,782 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:24:35,783 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:24:35,783 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:24:35,783 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:24:35,968 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:24:37,968 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:24:39,784 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:24:41,622 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:24:41,622 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:24:41,622 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:24:41,623 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:24:41,970 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:24:43,970 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:24:44,197 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:24:45,198 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:24:47,504 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:24:47,505 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:24:47,505 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:24:47,506 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:24:47,971 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:24:48,128 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:24:48,129 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:24:49,972 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:24:50,275 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:24:53,430 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:24:53,431 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:24:53,431 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:24:53,431 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:24:53,973 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:24:53,973 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:24:55,431 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:24:59,306 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:24:59,307 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:24:59,307 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:24:59,307 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:24:59,974 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:24:59,975 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:25:01,308 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:25:03,128 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:25:03,129 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:25:05,242 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:25:05,243 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:25:05,243 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:25:05,243 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:25:05,976 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:25:05,976 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:25:07,243 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:25:11,085 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:25:11,086 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:25:11,086 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:25:11,086 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:25:11,977 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:25:11,978 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:25:13,086 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:25:14,198 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:25:16,962 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:25:16,963 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:25:16,963 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:25:16,963 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:25:16,979 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:25:17,979 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:25:18,129 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:25:18,129 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:25:18,129 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:25:22,847 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:25:22,848 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:25:22,848 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:25:22,848 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:25:22,981 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:25:23,848 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:25:23,981 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:25:28,741 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:25:28,741 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:25:28,741 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:25:28,742 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:25:28,982 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:25:29,742 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:25:29,983 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:25:33,129 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:25:33,129 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:25:34,674 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:25:34,675 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:25:34,675 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:25:34,675 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:25:34,984 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:25:35,675 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:25:35,984 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:25:40,516 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:25:40,516 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:25:40,516 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:25:40,517 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:25:40,986 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:25:41,517 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:25:41,986 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:25:44,199 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:25:46,401 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:25:46,402 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:25:46,402 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:25:46,402 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:25:46,988 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:25:47,402 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:25:47,988 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:25:48,129 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:25:48,129 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:25:52,330 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:25:52,331 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:25:52,331 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:25:52,331 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:25:52,990 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:25:53,331 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:25:53,990 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:25:58,176 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:25:58,176 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:25:58,176 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:25:58,176 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:25:58,991 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:25:59,177 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:25:59,992 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:26:03,129 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:26:03,129 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:26:04,134 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:26:04,135 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:26:04,135 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:26:04,135 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:26:04,993 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:26:05,135 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:26:05,993 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:26:09,981 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:26:09,985 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:26:09,986 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:26:09,986 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:26:09,994 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:26:10,986 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:26:11,995 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:26:14,199 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:26:16,200 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:26:21,200 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:26:24,156 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:26:24,157 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:26:26,333 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:26:28,143 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:26:28,143 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:26:28,144 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:26:28,144 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:26:28,999 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:26:30,000 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:26:32,144 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:26:33,974 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:26:33,975 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:26:33,975 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:26:33,975 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:26:34,001 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:26:36,001 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:26:37,976 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:26:39,156 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:26:39,156 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:26:39,862 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:26:39,863 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:26:39,863 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:26:39,863 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:26:40,002 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:26:42,003 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:26:43,864 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:26:44,200 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:26:45,708 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:26:45,708 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:26:45,708 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:26:45,709 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:26:46,004 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:26:48,005 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:26:49,709 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:26:51,576 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:26:51,577 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:26:51,577 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:26:51,577 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:26:52,006 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:26:52,006 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:26:54,156 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:26:54,157 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:26:55,341 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:26:57,510 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:26:57,511 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:26:57,511 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:26:57,511 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:26:58,008 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:26:58,008 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:27:00,512 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:27:03,357 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:27:03,358 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:27:03,358 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:27:03,358 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:27:04,010 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:27:04,010 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:27:06,359 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:27:09,156 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:27:09,157 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:27:09,278 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:27:09,391 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:27:09,391 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:27:09,391 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:27:10,011 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:27:10,012 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:27:11,392 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:27:14,200 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:27:15,118 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:27:15,119 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:27:15,119 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:27:15,119 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:27:16,013 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:27:16,013 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:27:17,119 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:27:21,008 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:27:21,009 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:27:21,009 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:27:21,009 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:27:21,014 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:27:22,015 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:27:23,010 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:27:24,156 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:27:24,157 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:27:26,926 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:27:26,927 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:27:26,927 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:27:26,927 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:27:27,016 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:27:28,016 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:27:28,927 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:27:32,778 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:27:32,779 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:27:32,779 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:27:32,779 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:27:33,018 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:27:34,018 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:27:34,779 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:27:38,669 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:27:38,670 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:27:38,670 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:27:38,670 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:27:39,020 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:27:39,156 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:27:39,157 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:27:40,020 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:27:40,384 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:27:44,200 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:27:44,622 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:27:44,622 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:27:44,623 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:27:44,623 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:27:45,021 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:27:45,623 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:27:46,022 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:27:50,470 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:27:50,471 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:27:50,471 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:27:50,471 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:27:51,023 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:27:51,471 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:27:52,023 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:27:54,156 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:27:54,157 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:27:56,426 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:27:56,427 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:27:56,427 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:27:56,427 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:27:57,025 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:27:57,427 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:27:58,025 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:28:02,298 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:28:02,298 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:28:02,298 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:28:02,298 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:28:03,026 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:28:03,299 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:28:04,026 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:28:08,183 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:28:08,184 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:28:08,184 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:28:08,185 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:28:09,028 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:28:09,157 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:28:09,157 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:28:09,157 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:28:10,028 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:28:14,122 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:28:14,123 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:28:14,123 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:28:14,123 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:28:14,201 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:28:14,201 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:28:15,029 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:28:16,030 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:28:19,202 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:28:19,972 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:28:19,972 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:28:19,972 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:28:19,973 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:28:20,031 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:28:22,031 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:28:24,157 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:28:24,157 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:28:24,389 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:28:25,902 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:28:25,902 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:28:25,903 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:28:25,903 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:28:26,032 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:28:28,033 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:28:29,904 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:28:31,782 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:28:31,783 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:28:31,783 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:28:31,783 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:28:32,034 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:28:34,034 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:28:35,784 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:28:37,676 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:28:37,677 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:28:37,677 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:28:37,677 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:28:38,035 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:28:38,035 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:28:39,157 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:28:39,157 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:28:41,384 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:28:43,602 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:28:43,602 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:28:43,603 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:28:43,603 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:28:44,036 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:28:44,037 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:28:44,201 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:28:47,202 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:28:49,449 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:28:49,449 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:28:49,450 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:28:49,450 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:28:50,038 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:28:50,038 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:28:52,450 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:28:54,157 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:28:54,157 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:28:55,382 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:28:55,382 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:28:55,383 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:28:55,383 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:28:56,039 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:28:56,040 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:28:58,384 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:29:01,238 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:29:01,239 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:29:01,239 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:29:01,239 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:29:02,041 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:29:02,041 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:29:04,240 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:29:07,127 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:29:07,127 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:29:07,128 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:29:07,128 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:29:08,042 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:29:08,043 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:29:09,157 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:29:09,157 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:29:09,378 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:29:13,066 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:29:13,066 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:29:13,067 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:29:13,067 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:29:14,044 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:29:14,044 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:29:14,202 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:29:15,203 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:29:18,906 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: partial_history
2024-05-25 21:29:18,907 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:29:18,907 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:29:18,907 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:29:19,045 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:29:20,046 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:29:20,912 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:29:24,169 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: stop_status
2024-05-25 21:29:24,169 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: stop_status
2024-05-25 21:29:26,388 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:29:31,389 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:29:34,049 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:29:34,196 DEBUG   SenderThread:33547 [sender.py:send():378] send: telemetry
2024-05-25 21:29:34,197 DEBUG   SenderThread:33547 [sender.py:send():378] send: exit
2024-05-25 21:29:34,197 INFO    SenderThread:33547 [sender.py:send_exit():585] handling exit code: 0
2024-05-25 21:29:34,197 INFO    SenderThread:33547 [sender.py:send_exit():587] handling runtime: 3800
2024-05-25 21:29:34,197 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:29:34,197 INFO    SenderThread:33547 [sender.py:send_exit():593] send defer
2024-05-25 21:29:34,197 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: defer
2024-05-25 21:29:34,198 INFO    HandlerThread:33547 [handler.py:handle_request_defer():184] handle defer: 0
2024-05-25 21:29:34,198 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: defer
2024-05-25 21:29:34,198 INFO    SenderThread:33547 [sender.py:send_request_defer():609] handle sender defer: 0
2024-05-25 21:29:34,198 INFO    SenderThread:33547 [sender.py:transition_state():613] send defer: 1
2024-05-25 21:29:34,198 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: defer
2024-05-25 21:29:34,198 INFO    HandlerThread:33547 [handler.py:handle_request_defer():184] handle defer: 1
2024-05-25 21:29:34,198 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: defer
2024-05-25 21:29:34,198 INFO    SenderThread:33547 [sender.py:send_request_defer():609] handle sender defer: 1
2024-05-25 21:29:34,198 INFO    SenderThread:33547 [sender.py:transition_state():613] send defer: 2
2024-05-25 21:29:34,198 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: defer
2024-05-25 21:29:34,198 INFO    HandlerThread:33547 [handler.py:handle_request_defer():184] handle defer: 2
2024-05-25 21:29:34,198 INFO    HandlerThread:33547 [system_monitor.py:finish():203] Stopping system monitor
2024-05-25 21:29:34,198 DEBUG   SystemMonitor:33547 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-05-25 21:29:34,198 DEBUG   SystemMonitor:33547 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-05-25 21:29:34,199 INFO    HandlerThread:33547 [interfaces.py:finish():200] Joined cpu monitor
2024-05-25 21:29:34,199 INFO    HandlerThread:33547 [interfaces.py:finish():200] Joined disk monitor
2024-05-25 21:29:34,222 INFO    HandlerThread:33547 [interfaces.py:finish():200] Joined gpu monitor
2024-05-25 21:29:34,222 INFO    HandlerThread:33547 [interfaces.py:finish():200] Joined memory monitor
2024-05-25 21:29:34,222 INFO    HandlerThread:33547 [interfaces.py:finish():200] Joined network monitor
2024-05-25 21:29:34,223 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: defer
2024-05-25 21:29:34,223 INFO    SenderThread:33547 [sender.py:send_request_defer():609] handle sender defer: 2
2024-05-25 21:29:34,223 INFO    SenderThread:33547 [sender.py:transition_state():613] send defer: 3
2024-05-25 21:29:34,223 DEBUG   SenderThread:33547 [sender.py:send():378] send: stats
2024-05-25 21:29:34,223 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: defer
2024-05-25 21:29:34,223 INFO    HandlerThread:33547 [handler.py:handle_request_defer():184] handle defer: 3
2024-05-25 21:29:34,224 DEBUG   SenderThread:33547 [sender.py:send():378] send: history
2024-05-25 21:29:34,224 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: summary_record
2024-05-25 21:29:34,255 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:29:34,256 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: defer
2024-05-25 21:29:34,256 INFO    SenderThread:33547 [sender.py:send_request_defer():609] handle sender defer: 3
2024-05-25 21:29:34,256 INFO    SenderThread:33547 [sender.py:transition_state():613] send defer: 4
2024-05-25 21:29:34,256 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: defer
2024-05-25 21:29:34,256 INFO    HandlerThread:33547 [handler.py:handle_request_defer():184] handle defer: 4
2024-05-25 21:29:34,256 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: defer
2024-05-25 21:29:34,256 INFO    SenderThread:33547 [sender.py:send_request_defer():609] handle sender defer: 4
2024-05-25 21:29:34,256 INFO    SenderThread:33547 [sender.py:transition_state():613] send defer: 5
2024-05-25 21:29:34,256 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: defer
2024-05-25 21:29:34,256 INFO    HandlerThread:33547 [handler.py:handle_request_defer():184] handle defer: 5
2024-05-25 21:29:34,256 DEBUG   SenderThread:33547 [sender.py:send():378] send: summary
2024-05-25 21:29:34,257 INFO    SenderThread:33547 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end
2024-05-25 21:29:34,257 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: defer
2024-05-25 21:29:34,257 INFO    SenderThread:33547 [sender.py:send_request_defer():609] handle sender defer: 5
2024-05-25 21:29:34,257 INFO    SenderThread:33547 [sender.py:transition_state():613] send defer: 6
2024-05-25 21:29:34,257 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: defer
2024-05-25 21:29:34,257 INFO    HandlerThread:33547 [handler.py:handle_request_defer():184] handle defer: 6
2024-05-25 21:29:34,257 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: defer
2024-05-25 21:29:34,257 INFO    SenderThread:33547 [sender.py:send_request_defer():609] handle sender defer: 6
2024-05-25 21:29:34,259 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: status_report
2024-05-25 21:29:34,406 INFO    SenderThread:33547 [sender.py:transition_state():613] send defer: 7
2024-05-25 21:29:34,406 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: defer
2024-05-25 21:29:34,407 INFO    HandlerThread:33547 [handler.py:handle_request_defer():184] handle defer: 7
2024-05-25 21:29:34,407 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: defer
2024-05-25 21:29:34,407 INFO    SenderThread:33547 [sender.py:send_request_defer():609] handle sender defer: 7
2024-05-25 21:29:35,049 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/config.yaml
2024-05-25 21:29:35,050 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:29:35,197 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: poll_exit
2024-05-25 21:29:35,980 INFO    SenderThread:33547 [sender.py:transition_state():613] send defer: 8
2024-05-25 21:29:35,980 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: poll_exit
2024-05-25 21:29:35,981 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: defer
2024-05-25 21:29:35,981 INFO    HandlerThread:33547 [handler.py:handle_request_defer():184] handle defer: 8
2024-05-25 21:29:35,981 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: defer
2024-05-25 21:29:35,981 INFO    SenderThread:33547 [sender.py:send_request_defer():609] handle sender defer: 8
2024-05-25 21:29:35,981 INFO    SenderThread:33547 [job_builder.py:build():432] Attempting to build job artifact
2024-05-25 21:29:35,982 INFO    SenderThread:33547 [job_builder.py:_get_source_type():565] is repo sourced job
2024-05-25 21:29:35,982 WARNING SenderThread:33547 [job_builder.py:_log_if_verbose():267] No program path found, not creating job artifact. See https://docs.wandb.ai/guides/launch/create-job
2024-05-25 21:29:35,982 INFO    SenderThread:33547 [sender.py:transition_state():613] send defer: 9
2024-05-25 21:29:35,982 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: defer
2024-05-25 21:29:35,982 INFO    HandlerThread:33547 [handler.py:handle_request_defer():184] handle defer: 9
2024-05-25 21:29:35,982 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: defer
2024-05-25 21:29:35,982 INFO    SenderThread:33547 [sender.py:send_request_defer():609] handle sender defer: 9
2024-05-25 21:29:35,982 INFO    SenderThread:33547 [dir_watcher.py:finish():358] shutting down directory watcher
2024-05-25 21:29:36,050 INFO    Thread-12 :33547 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:29:36,050 INFO    SenderThread:33547 [dir_watcher.py:finish():388] scan: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files
2024-05-25 21:29:36,050 INFO    SenderThread:33547 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/conda-environment.yaml conda-environment.yaml
2024-05-25 21:29:36,050 INFO    SenderThread:33547 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log output.log
2024-05-25 21:29:36,050 INFO    SenderThread:33547 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-metadata.json wandb-metadata.json
2024-05-25 21:29:36,051 INFO    SenderThread:33547 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/config.yaml config.yaml
2024-05-25 21:29:36,051 INFO    SenderThread:33547 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/requirements.txt requirements.txt
2024-05-25 21:29:36,051 INFO    SenderThread:33547 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json wandb-summary.json
2024-05-25 21:29:36,052 INFO    SenderThread:33547 [sender.py:transition_state():613] send defer: 10
2024-05-25 21:29:36,052 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: defer
2024-05-25 21:29:36,053 INFO    HandlerThread:33547 [handler.py:handle_request_defer():184] handle defer: 10
2024-05-25 21:29:36,054 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: defer
2024-05-25 21:29:36,054 INFO    SenderThread:33547 [sender.py:send_request_defer():609] handle sender defer: 10
2024-05-25 21:29:36,054 INFO    SenderThread:33547 [file_pusher.py:finish():169] shutting down file pusher
2024-05-25 21:29:36,197 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: poll_exit
2024-05-25 21:29:36,198 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: poll_exit
2024-05-25 21:29:36,414 INFO    wandb-upload_2:33547 [upload_job.py:push():130] Uploaded file /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/config.yaml
2024-05-25 21:29:36,415 INFO    wandb-upload_3:33547 [upload_job.py:push():130] Uploaded file /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/requirements.txt
2024-05-25 21:29:36,428 INFO    wandb-upload_0:33547 [upload_job.py:push():130] Uploaded file /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/conda-environment.yaml
2024-05-25 21:29:36,449 INFO    wandb-upload_4:33547 [upload_job.py:push():130] Uploaded file /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/wandb-summary.json
2024-05-25 21:29:36,661 INFO    wandb-upload_1:33547 [upload_job.py:push():130] Uploaded file /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/files/output.log
2024-05-25 21:29:36,862 INFO    Thread-11 (_thread_body):33547 [sender.py:transition_state():613] send defer: 11
2024-05-25 21:29:36,862 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: defer
2024-05-25 21:29:36,862 INFO    HandlerThread:33547 [handler.py:handle_request_defer():184] handle defer: 11
2024-05-25 21:29:36,863 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: defer
2024-05-25 21:29:36,863 INFO    SenderThread:33547 [sender.py:send_request_defer():609] handle sender defer: 11
2024-05-25 21:29:36,863 INFO    SenderThread:33547 [file_pusher.py:join():175] waiting for file pusher
2024-05-25 21:29:36,863 INFO    SenderThread:33547 [sender.py:transition_state():613] send defer: 12
2024-05-25 21:29:36,863 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: defer
2024-05-25 21:29:36,863 INFO    HandlerThread:33547 [handler.py:handle_request_defer():184] handle defer: 12
2024-05-25 21:29:36,863 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: defer
2024-05-25 21:29:36,863 INFO    SenderThread:33547 [sender.py:send_request_defer():609] handle sender defer: 12
2024-05-25 21:29:36,863 INFO    SenderThread:33547 [file_stream.py:finish():601] file stream finish called
2024-05-25 21:29:37,198 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: poll_exit
2024-05-25 21:29:37,310 INFO    SenderThread:33547 [file_stream.py:finish():605] file stream finish is done
2024-05-25 21:29:37,310 INFO    SenderThread:33547 [sender.py:transition_state():613] send defer: 13
2024-05-25 21:29:37,310 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: poll_exit
2024-05-25 21:29:37,310 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: defer
2024-05-25 21:29:37,310 INFO    HandlerThread:33547 [handler.py:handle_request_defer():184] handle defer: 13
2024-05-25 21:29:37,310 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: defer
2024-05-25 21:29:37,311 INFO    SenderThread:33547 [sender.py:send_request_defer():609] handle sender defer: 13
2024-05-25 21:29:37,311 INFO    SenderThread:33547 [sender.py:transition_state():613] send defer: 14
2024-05-25 21:29:37,311 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: defer
2024-05-25 21:29:37,311 INFO    HandlerThread:33547 [handler.py:handle_request_defer():184] handle defer: 14
2024-05-25 21:29:37,311 DEBUG   SenderThread:33547 [sender.py:send():378] send: final
2024-05-25 21:29:37,311 DEBUG   SenderThread:33547 [sender.py:send():378] send: footer
2024-05-25 21:29:37,311 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: defer
2024-05-25 21:29:37,311 INFO    SenderThread:33547 [sender.py:send_request_defer():609] handle sender defer: 14
2024-05-25 21:29:37,311 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: poll_exit
2024-05-25 21:29:37,312 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: poll_exit
2024-05-25 21:29:37,312 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: internal_messages
2024-05-25 21:29:37,312 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: server_info
2024-05-25 21:29:37,312 DEBUG   SenderThread:33547 [sender.py:send_request():405] send_request: server_info
2024-05-25 21:29:37,354 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: get_summary
2024-05-25 21:29:37,354 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: sampled_history
2024-05-25 21:29:37,443 DEBUG   HandlerThread:33547 [handler.py:handle_request():158] handle_request: shutdown
2024-05-25 21:29:37,443 INFO    HandlerThread:33547 [handler.py:finish():882] shutting down handler
2024-05-25 21:29:38,312 INFO    WriterThread:33547 [datastore.py:close():296] close: /root/mistral-finetune/runs2/wandb/run-20240525_202613-sapihyp4/run-sapihyp4.wandb
2024-05-25 21:29:38,443 INFO    SenderThread:33547 [sender.py:finish():1545] shutting down sender
2024-05-25 21:29:38,443 INFO    SenderThread:33547 [file_pusher.py:finish():169] shutting down file pusher
2024-05-25 21:29:38,443 INFO    SenderThread:33547 [file_pusher.py:join():175] waiting for file pusher