2024-05-25 21:46:43,432 INFO StreamThr :43320 [internal.py:wandb_internal():85] W&B internal server running at pid: 43320, started at: 2024-05-25 21:46:43.432097 2024-05-25 21:46:43,433 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status 2024-05-25 21:46:43,434 INFO WriterThread:43320 [datastore.py:open_for_write():87] open: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/run-cngfo6vp.wandb 2024-05-25 21:46:43,436 DEBUG SenderThread:43320 [sender.py:send():378] send: header 2024-05-25 21:46:43,436 DEBUG SenderThread:43320 [sender.py:send():378] send: run 2024-05-25 21:46:43,823 INFO SenderThread:43320 [dir_watcher.py:__init__():211] watching files in: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files 2024-05-25 21:46:43,823 INFO SenderThread:43320 [sender.py:_start_run_threads():1123] run started: cngfo6vp with start time 1716673603.431967 2024-05-25 21:46:43,827 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: check_version 2024-05-25 21:46:43,827 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: check_version 2024-05-25 21:46:43,906 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: run_start 2024-05-25 21:46:43,911 DEBUG HandlerThread:43320 [system_info.py:__init__():26] System info init 2024-05-25 21:46:43,911 DEBUG HandlerThread:43320 [system_info.py:__init__():41] System info init done 2024-05-25 21:46:43,911 INFO HandlerThread:43320 [system_monitor.py:start():194] Starting system monitor 2024-05-25 21:46:43,911 INFO SystemMonitor:43320 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-05-25 21:46:43,912 INFO HandlerThread:43320 [system_monitor.py:probe():214] Collecting system info 2024-05-25 21:46:43,912 INFO SystemMonitor:43320 [interfaces.py:start():188] Started cpu monitoring 2024-05-25 21:46:43,913 INFO SystemMonitor:43320 [interfaces.py:start():188] Started disk monitoring 2024-05-25 21:46:43,913 INFO SystemMonitor:43320 [interfaces.py:start():188] Started gpu monitoring 2024-05-25 21:46:43,914 INFO SystemMonitor:43320 [interfaces.py:start():188] Started memory monitoring 2024-05-25 21:46:43,915 INFO SystemMonitor:43320 [interfaces.py:start():188] Started network monitoring 2024-05-25 21:46:43,923 DEBUG HandlerThread:43320 [system_info.py:probe():150] Probing system 2024-05-25 21:46:43,924 DEBUG HandlerThread:43320 [system_info.py:_probe_git():135] Probing git 2024-05-25 21:46:43,928 DEBUG HandlerThread:43320 [system_info.py:_probe_git():143] Probing git done 2024-05-25 21:46:43,928 DEBUG HandlerThread:43320 [system_info.py:probe():198] Probing system done 2024-05-25 21:46:43,928 DEBUG HandlerThread:43320 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-102-generic-x86_64-with-glibc2.35', 'python': '3.10.14', 'heartbeatAt': '2024-05-25T21:46:43.923639', 'startedAt': '2024-05-25T21:46:43.429708', 'docker': None, 'cuda': None, 'args': ('example/7B.yaml',), 'state': 'running', 'program': '-m train', 'codePathLocal': None, 'git': {'remote': 'https://github.com/mistralai/mistral-finetune.git', 'commit': 'cb892f012ef940bc741dd3b043bc1d06f0cf6085'}, 'email': 'clement.castellon@gmail.com', 'root': '/root/mistral-finetune', 'host': 'scw-tender-archimedes', 'username': 'root', 'executable': '/root/miniconda/envs/chemistralpy310/bin/python', 'cpu_count': 24, 'cpu_count_logical': 24, 'cpu_freq': {'current': 2699.9959999999996, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}, {'current': 2699.996, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 902.4293403625488, 'used': 190.7148666381836}}, 'gpu': 'NVIDIA H100 PCIe', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA H100 PCIe', 'memory_total': 85520809984}], 'memory': {'total': 235.9408187866211}} 2024-05-25 21:46:43,928 INFO HandlerThread:43320 [system_monitor.py:probe():224] Finished collecting system info 2024-05-25 21:46:43,928 INFO HandlerThread:43320 [system_monitor.py:probe():227] Publishing system info 2024-05-25 21:46:43,928 DEBUG HandlerThread:43320 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment 2024-05-25 21:46:44,823 INFO Thread-12 :43320 [dir_watcher.py:_on_file_created():271] file/dir created: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/conda-environment.yaml 2024-05-25 21:46:44,893 DEBUG HandlerThread:43320 [system_info.py:_save_conda():222] Saving conda packages done 2024-05-25 21:46:44,893 INFO HandlerThread:43320 [system_monitor.py:probe():229] Finished publishing system info 2024-05-25 21:46:44,896 DEBUG SenderThread:43320 [sender.py:send():378] send: files 2024-05-25 21:46:44,896 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-metadata.json with policy now 2024-05-25 21:46:44,923 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: python_packages 2024-05-25 21:46:44,923 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: python_packages 2024-05-25 21:46:44,923 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:46:44,923 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:46:45,112 DEBUG SenderThread:43320 [sender.py:send():378] send: telemetry 2024-05-25 21:46:45,371 INFO wandb-upload_0:43320 [upload_job.py:push():130] Uploaded file /tmp/tmpr2ktm50cwandb/nymq5ayo-wandb-metadata.json 2024-05-25 21:46:45,823 INFO Thread-12 :43320 [dir_watcher.py:_on_file_created():271] file/dir created: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-metadata.json 2024-05-25 21:46:45,823 INFO Thread-12 :43320 [dir_watcher.py:_on_file_created():271] file/dir created: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/requirements.txt 2024-05-25 21:46:45,823 INFO Thread-12 :43320 [dir_watcher.py:_on_file_created():271] file/dir created: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 21:46:47,824 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 21:46:49,012 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:46:54,012 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:46:59,013 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:46:59,944 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:46:59,944 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:47:04,168 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:47:09,168 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:47:14,171 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:47:14,830 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/config.yaml 2024-05-25 21:47:14,944 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:47:14,945 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:47:20,129 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:47:25,130 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:47:29,965 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:47:29,966 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:47:31,122 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:47:36,122 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:47:41,123 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:47:43,915 DEBUG SystemMonitor:43320 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-05-25 21:47:43,916 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:47:44,982 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:47:44,982 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:47:47,116 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:47:52,117 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:47:57,118 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:47:59,982 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:47:59,982 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:48:02,186 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:48:07,186 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:48:12,187 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:48:13,916 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:48:14,982 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:48:14,982 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:48:18,164 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:48:23,165 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:48:28,166 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:48:29,977 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:48:29,978 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:48:33,172 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:48:38,173 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:48:43,174 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:48:43,917 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:48:45,029 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:48:45,029 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:48:48,259 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:48:53,259 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:48:58,260 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:49:00,009 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:49:00,009 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:49:04,182 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:49:09,183 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:49:13,918 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:49:14,918 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:49:15,035 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:49:15,035 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:49:20,241 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:49:25,242 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:49:30,040 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:49:30,041 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:49:31,180 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:49:36,181 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:49:41,182 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:49:43,918 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:49:45,053 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:49:45,054 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:49:46,276 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:49:51,277 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:49:56,277 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:50:00,062 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:50:00,062 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:50:01,280 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:50:06,281 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:50:11,282 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:50:13,919 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:50:15,062 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:50:15,062 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:50:16,295 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:50:21,296 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:50:26,296 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:50:30,052 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:50:30,052 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:50:32,216 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:50:37,216 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:50:42,217 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:50:43,919 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:50:45,094 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:50:45,094 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:50:47,276 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:50:52,276 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:50:57,277 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:51:00,083 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:51:00,084 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:51:03,215 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:51:08,216 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:51:13,217 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:51:13,920 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:51:15,068 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:51:15,068 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:51:19,213 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:51:24,214 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:51:29,215 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:51:30,073 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:51:30,073 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:51:34,286 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:51:39,287 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:51:43,920 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:51:44,921 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:51:45,089 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:51:45,090 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:51:50,329 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:51:55,330 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:52:00,098 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:52:00,098 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:52:00,337 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:52:05,337 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:52:10,338 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:52:13,921 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:52:15,093 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:52:15,093 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:52:16,322 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:52:21,323 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:52:26,323 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:52:30,098 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:52:30,098 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:52:32,265 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:52:37,266 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:52:42,267 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:52:43,921 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:52:45,088 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:52:45,088 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:52:48,239 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:52:53,239 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:52:58,240 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:53:00,122 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:53:00,122 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:53:03,369 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:53:08,370 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:53:13,371 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:53:13,922 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:53:15,109 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:53:15,109 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:53:19,273 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:53:24,274 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:53:29,275 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:53:30,111 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:53:30,111 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:53:34,314 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:53:39,314 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:53:43,922 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:53:44,923 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:53:45,125 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:53:45,125 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:53:50,308 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:53:55,309 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:54:00,155 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:54:00,155 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:54:00,373 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:54:05,374 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:54:10,374 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:54:13,923 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:54:15,615 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:54:15,646 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:54:15,646 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:54:20,869 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:54:25,870 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:54:30,646 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:54:30,647 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:54:31,822 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:54:36,823 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:54:41,823 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:54:43,923 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:54:45,671 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:54:45,671 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:54:46,862 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:54:51,863 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:54:56,863 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:55:00,646 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:55:00,646 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:55:01,875 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:55:06,875 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:55:11,876 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:55:13,924 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:55:15,672 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:55:15,672 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:55:16,880 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:55:21,881 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:55:26,882 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:55:30,676 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:55:30,676 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:55:32,876 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:55:37,877 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:55:42,878 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:55:43,924 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:55:45,682 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:55:45,682 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:55:48,876 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:55:53,877 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:55:58,878 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:56:00,683 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:56:00,683 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:56:03,892 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:56:08,893 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:56:13,893 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:56:13,924 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:56:15,707 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:56:15,707 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:56:19,852 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:56:24,853 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:56:29,854 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:56:30,676 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:56:30,676 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:56:34,875 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:56:39,876 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:56:43,925 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:56:44,926 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:56:45,712 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:56:45,712 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:56:50,876 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:56:55,876 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:57:00,703 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:57:00,703 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:57:00,901 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:57:05,901 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:57:10,902 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:57:13,925 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:57:15,708 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:57:15,708 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:57:16,895 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:57:21,896 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:57:26,897 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:57:30,700 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:57:30,700 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:57:32,841 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:57:37,842 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:57:42,842 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:57:43,930 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:57:45,705 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:57:45,705 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:57:47,891 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:57:52,891 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:57:57,892 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:58:00,729 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:58:00,729 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:58:02,907 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:58:07,907 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:58:12,908 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:58:13,926 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:58:15,729 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:58:15,729 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:58:17,946 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:58:22,947 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:58:27,948 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:58:30,734 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:58:30,734 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:58:33,833 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:58:38,834 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:58:43,835 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:58:43,928 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:58:47,664 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:58:47,665 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:58:49,797 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:58:54,797 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:59:00,001 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 21:59:00,306 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:59:02,607 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:59:02,607 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:59:05,749 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:59:10,332 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 21:59:10,333 DEBUG SenderThread:43320 [sender.py:send():378] send: telemetry 2024-05-25 21:59:11,335 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:59:12,003 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/config.yaml 2024-05-25 21:59:12,004 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 21:59:13,928 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:59:16,928 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:59:17,607 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:59:17,607 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:59:21,950 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 21:59:21,951 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:59:21,951 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 21:59:21,951 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 21:59:21,951 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 21:59:22,006 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 21:59:22,006 INFO Thread-12 :43320 [dir_watcher.py:_on_file_created():271] file/dir created: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 21:59:26,952 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:59:31,953 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:59:32,607 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:59:32,607 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:59:33,692 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 21:59:33,693 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 21:59:33,693 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 21:59:33,693 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 21:59:34,009 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 21:59:34,009 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 21:59:37,694 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:59:42,695 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:59:43,928 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 21:59:45,462 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 21:59:45,463 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 21:59:45,463 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 21:59:45,463 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 21:59:46,011 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 21:59:47,012 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 21:59:47,607 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 21:59:47,607 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 21:59:47,745 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:59:52,746 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:59:57,257 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 21:59:57,258 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 21:59:57,258 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 21:59:57,258 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 21:59:58,014 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 21:59:58,259 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 21:59:59,014 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:00:02,607 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:00:02,607 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:00:03,857 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:00:08,857 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:00:09,089 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:00:09,089 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:00:09,089 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:00:09,090 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:00:10,016 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:00:11,017 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:00:13,928 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:00:13,930 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:00:17,607 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:00:17,608 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:00:19,757 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:00:20,911 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:00:20,911 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:00:20,911 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:00:20,912 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:00:21,019 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:00:23,019 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:00:24,912 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:00:29,913 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:00:32,607 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:00:32,608 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:00:32,767 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:00:32,767 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:00:32,767 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:00:32,768 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:00:33,021 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:00:35,022 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:00:35,768 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:00:40,769 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:00:43,929 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:00:44,609 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:00:44,610 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:00:44,610 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:00:44,610 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:00:45,024 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:00:46,611 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:00:47,025 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:00:47,607 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:00:47,608 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:00:51,818 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:00:56,435 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:00:56,436 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:00:56,436 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:00:56,436 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:00:57,027 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:00:57,437 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:00:59,027 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:01:02,437 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:01:02,607 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:01:02,608 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:01:07,838 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:01:08,273 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:01:08,274 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:01:08,274 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:01:08,274 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:01:09,030 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:01:11,031 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:01:13,275 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:01:13,929 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:01:17,608 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:01:17,608 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:01:18,785 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:01:20,104 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:01:20,105 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:01:20,105 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:01:20,105 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:01:21,033 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:01:21,033 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:01:24,106 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:01:29,107 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:01:31,936 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:01:31,936 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:01:31,937 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:01:31,937 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:01:32,036 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:01:32,608 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:01:32,608 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:01:33,036 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:01:34,749 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:01:39,750 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:01:43,768 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:01:43,769 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:01:43,769 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:01:43,769 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:01:43,930 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:01:44,039 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:01:44,930 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:01:45,039 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:01:47,608 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:01:47,608 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:01:50,813 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:01:55,624 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:01:55,625 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:01:55,625 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:01:55,625 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:01:56,042 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:01:56,626 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:01:57,042 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:02:01,626 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:02:02,608 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:02:02,609 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:02:06,768 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:02:07,474 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:02:07,475 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:02:07,475 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:02:07,475 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:02:08,045 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:02:09,046 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:02:12,476 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:02:13,930 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:02:17,608 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:02:17,608 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:02:17,825 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:02:19,312 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:02:19,312 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:02:19,312 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:02:19,313 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:02:20,048 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:02:21,049 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:02:23,314 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:02:28,314 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:02:31,168 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:02:31,169 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:02:31,169 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:02:31,169 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:02:32,051 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:02:32,608 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:02:32,608 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:02:33,052 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:02:33,833 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:02:38,834 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:02:43,045 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:02:43,045 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:02:43,046 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:02:43,046 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:02:43,054 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:02:43,931 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:02:43,931 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:02:45,055 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:02:47,608 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:02:47,609 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:02:49,759 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:02:54,760 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:02:54,906 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:02:54,906 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:02:54,906 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:02:54,907 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:02:55,057 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:02:57,058 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:02:59,908 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:03:02,608 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:03:02,609 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:03:05,749 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:03:06,757 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:03:06,758 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:03:06,758 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:03:06,758 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:03:07,060 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:03:09,061 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:03:10,759 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:03:13,931 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:03:15,932 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:03:17,608 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:03:17,609 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:03:18,575 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:03:18,588 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:03:18,588 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:03:18,588 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:03:19,063 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:03:21,064 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:03:21,589 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:03:26,590 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:03:30,410 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:03:30,410 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:03:30,411 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:03:30,411 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:03:31,066 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:03:32,411 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:03:32,609 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:03:32,609 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:03:33,066 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:03:37,814 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:03:42,241 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:03:42,242 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:03:42,242 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:03:42,242 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:03:43,069 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:03:43,069 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:03:43,243 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:03:43,932 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:03:47,609 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:03:47,609 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:03:48,831 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:03:53,831 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:03:54,060 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:03:54,061 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:03:54,061 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:03:54,061 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:03:54,071 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:03:55,071 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:03:59,062 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:04:02,609 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:04:02,609 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:04:04,816 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:04:05,884 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:04:05,884 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:04:05,885 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:04:05,885 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:04:06,074 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:04:07,074 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:04:09,886 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:04:13,932 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:04:14,933 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:04:17,609 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:04:17,609 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:04:17,715 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:04:17,808 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:04:17,809 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:04:17,809 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:04:18,077 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:04:19,077 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:04:20,810 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:04:25,810 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:04:29,545 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:04:29,546 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:04:29,546 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:04:29,546 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:04:30,080 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:04:31,080 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:04:31,547 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:04:32,609 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:04:32,609 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:04:36,846 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:04:41,383 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:04:41,383 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:04:41,384 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:04:41,384 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:04:42,083 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:04:42,384 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:04:43,083 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:04:43,933 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:04:47,609 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:04:47,609 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:04:47,841 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:04:52,842 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:04:53,227 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:04:53,228 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:04:53,228 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:04:53,228 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:04:54,086 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:04:55,087 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:04:58,229 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:05:02,609 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:05:02,609 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:05:03,744 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:05:05,075 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:05:05,075 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:05:05,075 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:05:05,076 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:05:05,089 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:05:07,090 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:05:09,076 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:05:13,933 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:05:14,934 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:05:16,930 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:05:16,931 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:05:16,931 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:05:16,931 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:05:17,092 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:05:17,609 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:05:17,609 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:05:19,093 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:05:20,763 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:05:25,764 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:05:28,782 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:05:28,783 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:05:28,783 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:05:28,783 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:05:29,095 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:05:30,784 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:05:31,096 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:05:32,609 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:05:32,610 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:05:36,769 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:05:40,596 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:05:40,597 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:05:40,597 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:05:40,597 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:05:41,098 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:05:42,598 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:05:43,098 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:05:43,934 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:05:47,609 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:05:47,610 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:05:47,816 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:05:52,413 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:05:52,414 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:05:52,414 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:05:52,414 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:05:53,101 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:05:53,415 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:05:55,101 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:05:58,415 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:06:02,610 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:06:02,610 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:06:03,782 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:06:04,243 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:06:04,243 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:06:04,244 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:06:04,244 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:06:05,104 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:06:05,104 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:06:09,245 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:06:13,934 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:06:14,935 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:06:16,065 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:06:16,066 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:06:16,066 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:06:16,067 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:06:16,107 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:06:17,107 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:06:17,610 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:06:17,610 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:06:20,795 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:06:25,795 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:06:27,878 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:06:27,879 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:06:27,879 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:06:27,879 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:06:28,110 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:06:29,110 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:06:30,880 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:06:32,610 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:06:32,610 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:06:36,847 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:06:39,718 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:06:39,719 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:06:39,719 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:06:39,719 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:06:40,113 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:06:41,113 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:06:42,720 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:06:43,935 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:06:47,610 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:06:47,610 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:06:47,836 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:06:51,557 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:06:51,557 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:06:51,557 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:06:51,558 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:06:52,116 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:06:53,116 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:06:53,558 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:06:58,559 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:07:02,610 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:07:02,610 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:07:03,393 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:07:03,394 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:07:03,394 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:07:03,394 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:07:04,118 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:07:04,395 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:07:05,119 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:07:09,395 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:07:13,935 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:07:14,936 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:07:15,246 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:07:15,247 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:07:15,247 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:07:15,247 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:07:16,121 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:07:17,122 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:07:17,610 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:07:17,611 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:07:20,793 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:07:25,794 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:07:27,066 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:07:27,067 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:07:27,067 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:07:27,067 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:07:27,124 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:07:29,124 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:07:31,068 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:07:32,610 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:07:32,611 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:07:36,800 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:07:38,896 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:07:38,896 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:07:38,897 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:07:38,897 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:07:39,127 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:07:41,127 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:07:41,897 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:07:43,936 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:07:46,937 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:07:47,610 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:07:47,611 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:07:50,745 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:07:50,745 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:07:50,745 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:07:50,746 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:07:51,130 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:07:52,746 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:07:53,130 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:07:57,747 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:08:02,604 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:08:02,605 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:08:02,605 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:08:02,606 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:08:02,615 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:08:02,615 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:08:02,839 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:08:03,132 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:08:05,133 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:08:07,839 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:08:12,840 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:08:13,940 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:08:14,436 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:08:14,437 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:08:14,437 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:08:14,438 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:08:15,135 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:08:15,135 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:08:17,615 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:08:17,615 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:08:17,856 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:08:22,856 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:08:26,271 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:08:26,272 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:08:26,272 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:08:26,272 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:08:27,138 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:08:27,138 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:08:28,272 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:08:32,615 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:08:32,616 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:08:33,751 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:08:38,144 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:08:38,144 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:08:38,144 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:08:38,144 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:08:39,140 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:08:39,140 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:08:39,145 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:08:43,937 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:08:44,937 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:08:47,615 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:08:47,616 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:08:49,974 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:08:49,974 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:08:49,975 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:08:49,975 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:08:49,976 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:08:50,142 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:08:51,143 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:08:54,976 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:08:59,977 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:09:01,800 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:09:01,801 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:09:01,801 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:09:01,801 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:09:02,145 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:09:02,615 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:09:02,616 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:09:03,145 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:09:05,810 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:09:10,811 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:09:13,639 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:09:13,640 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:09:13,640 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:09:13,640 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:09:13,937 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:09:14,147 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:09:15,148 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:09:15,938 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:09:17,616 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:09:17,616 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:09:21,768 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:09:25,515 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:09:25,516 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:09:25,516 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:09:25,517 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:09:26,150 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:09:27,150 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:09:27,517 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:09:32,518 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:09:32,616 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:09:32,616 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:09:37,368 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:09:37,368 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:09:37,369 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:09:37,369 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:09:38,153 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:09:38,369 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:09:39,153 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:09:43,370 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:09:43,938 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:09:47,616 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:09:47,616 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:09:48,834 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:09:49,212 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:09:49,213 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:09:49,213 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:09:49,213 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:09:50,156 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:09:51,156 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:09:54,214 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:09:59,214 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:10:01,055 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:10:01,056 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:10:01,056 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:10:01,057 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:10:01,159 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:10:02,616 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:10:02,616 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:10:03,159 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:10:04,771 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:10:09,772 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:10:12,920 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:10:12,920 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:10:12,920 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:10:12,921 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:10:13,161 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:10:13,938 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:10:14,939 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:10:15,162 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:10:17,616 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:10:17,616 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:10:20,832 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:10:24,751 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:10:24,752 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:10:24,752 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:10:24,752 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:10:25,164 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:10:26,753 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:10:27,165 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:10:31,753 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:10:32,616 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:10:32,617 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:10:36,613 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:10:36,614 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:10:36,614 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:10:36,615 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:10:37,167 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:10:37,167 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:10:37,615 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:10:42,616 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:10:43,939 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:10:47,616 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:10:47,617 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:10:47,741 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:10:48,473 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:10:48,473 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:10:48,473 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:10:48,474 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:10:49,170 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:10:49,170 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:10:53,475 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:10:58,475 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:11:00,320 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:11:00,321 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:11:00,321 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:11:00,321 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:11:01,173 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:11:01,173 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:11:02,616 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:11:02,617 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:11:03,803 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:11:08,804 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:11:12,166 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:11:12,166 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:11:12,167 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:11:12,168 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:11:12,176 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:11:13,176 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:11:13,939 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:11:13,940 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:11:17,617 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:11:17,617 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:11:19,778 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:11:24,001 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:11:24,002 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:11:24,002 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:11:24,003 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:11:24,178 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:11:25,003 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:11:25,179 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:11:30,004 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:11:32,617 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:11:32,617 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:11:35,795 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:11:35,843 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:11:35,843 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:11:35,844 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:11:35,844 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:11:36,181 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:11:37,182 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:11:40,845 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:11:43,940 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:11:45,940 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:11:47,617 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:11:47,617 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:11:47,687 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:11:47,827 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:11:47,827 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:11:47,827 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:11:48,184 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:11:49,184 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:11:51,828 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:11:56,829 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:11:59,523 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:11:59,523 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:11:59,524 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:11:59,524 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:12:00,187 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:12:01,187 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:12:02,525 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:12:02,617 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:12:02,617 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:12:07,778 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:12:11,357 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:12:11,358 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:12:11,358 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:12:11,358 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:12:12,190 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:12:13,190 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:12:13,359 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:12:13,940 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:12:17,617 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:12:17,617 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:12:18,782 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:12:23,204 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:12:23,205 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:12:23,205 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:12:23,205 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:12:24,193 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:12:24,206 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:12:25,193 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:12:29,206 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:12:32,617 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:12:32,618 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:12:34,779 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:12:35,053 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:12:35,054 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:12:35,054 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:12:35,054 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:12:35,196 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:12:37,196 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:12:40,055 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:12:43,941 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:12:45,941 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:12:46,908 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:12:46,908 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:12:46,908 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:12:46,909 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:12:47,198 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:12:47,617 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:12:47,618 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:12:49,199 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:12:51,849 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:12:56,849 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:12:58,749 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:12:58,750 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:12:58,750 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:12:58,750 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:12:59,201 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:12:59,202 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:13:02,617 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:13:02,618 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:13:02,796 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:13:07,797 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:13:10,583 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:13:10,584 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:13:10,584 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:13:10,584 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:13:11,204 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:13:11,204 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:13:13,585 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:13:13,945 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:13:17,618 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:13:17,618 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:13:18,772 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:13:22,425 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:13:22,425 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:13:22,426 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:13:22,426 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:13:23,207 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:13:23,207 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:13:24,426 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:13:29,427 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:13:32,618 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:13:32,618 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:13:34,269 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:13:34,269 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:13:34,270 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:13:34,270 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:13:35,210 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:13:35,210 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:13:35,270 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:13:40,271 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:13:43,942 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:13:45,942 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:13:46,083 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:13:46,083 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:13:46,084 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:13:46,085 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:13:46,213 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:13:47,213 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:13:47,618 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:13:47,618 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:13:51,751 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:13:56,751 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:13:57,949 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:13:57,950 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:13:57,950 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:13:57,950 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:13:58,216 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:13:59,216 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:14:01,951 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:14:02,618 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:14:02,618 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:14:07,824 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:14:09,783 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:14:09,784 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:14:09,784 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:14:09,784 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:14:10,219 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:14:11,219 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:14:13,785 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:14:13,942 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:14:17,618 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:14:17,619 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:14:19,771 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:14:21,638 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:14:21,639 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:14:21,639 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:14:21,640 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:14:22,222 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:14:23,222 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:14:25,641 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:14:30,641 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:14:32,618 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:14:32,619 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:14:33,513 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:14:33,514 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:14:33,514 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:14:33,515 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:14:34,225 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:14:35,225 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:14:36,515 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:14:41,516 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:14:43,943 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:14:45,391 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:14:45,391 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:14:45,392 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:14:45,392 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:14:46,228 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:14:47,228 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:14:47,392 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:14:47,618 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:14:47,619 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:14:52,782 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:14:57,232 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:14:57,232 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:14:57,233 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:14:57,233 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:14:58,230 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:14:58,234 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:14:59,231 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:15:02,618 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:15:02,619 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:15:03,827 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:15:08,827 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:15:09,081 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:15:09,082 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:15:09,082 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:15:09,082 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:15:09,233 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:15:11,233 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:15:13,943 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:15:13,944 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:15:17,619 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:15:17,619 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:15:19,786 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:15:20,916 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:15:20,916 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:15:20,917 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:15:20,917 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:15:21,236 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:15:21,236 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:15:24,918 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:15:29,918 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:15:32,619 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:15:32,619 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:15:32,775 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:15:32,841 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:15:32,842 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:15:32,843 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:15:33,239 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:15:33,239 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:15:35,844 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:15:40,844 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:15:43,944 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:15:44,623 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:15:44,624 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:15:44,624 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:15:44,624 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:15:45,242 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:15:45,242 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:15:46,625 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:15:47,619 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:15:47,619 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:15:51,777 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:15:56,485 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:15:56,486 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:15:56,486 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:15:56,486 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:15:57,245 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:15:57,245 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:15:57,486 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:16:02,487 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:16:02,619 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:16:02,619 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:16:07,828 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:16:08,326 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:16:08,326 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:16:08,326 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:16:08,327 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:16:09,248 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:16:09,249 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:16:13,328 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:16:13,944 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:16:17,619 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:16:17,619 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:16:18,805 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:16:20,164 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:16:20,164 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:16:20,165 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:16:20,165 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:16:20,251 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:16:21,251 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:16:24,165 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:16:29,166 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:16:32,018 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:16:32,019 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:16:32,019 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:16:32,019 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:16:32,254 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:16:32,619 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:16:32,619 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:16:33,254 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:16:34,779 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:16:39,779 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:16:43,871 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:16:43,872 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:16:43,872 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:16:43,873 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:16:43,945 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:16:44,256 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:16:44,945 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:16:45,257 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:16:47,619 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:16:47,620 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:16:50,851 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:16:55,721 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:16:55,722 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:16:55,722 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:16:55,722 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:16:56,259 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:16:56,722 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:16:57,259 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:17:01,723 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:17:02,619 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:17:02,620 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:17:06,850 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:17:07,640 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:17:07,641 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:17:07,641 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:17:07,641 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:17:08,262 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:17:09,262 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:17:12,642 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:17:13,945 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:17:17,620 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:17:17,620 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:17:17,849 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:17:19,500 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:17:19,500 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:17:19,500 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:17:19,501 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:17:20,265 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:17:21,265 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:17:23,501 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:17:28,502 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:17:31,339 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:17:31,340 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:17:31,340 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:17:31,340 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:17:32,268 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:17:32,620 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:17:32,620 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:17:33,761 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:17:34,268 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:17:38,762 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:17:43,182 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:17:43,183 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:17:43,183 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:17:43,183 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:17:43,270 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:17:43,946 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:17:43,947 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:17:44,271 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:17:47,620 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:17:47,620 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:17:49,815 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:17:54,816 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:17:55,059 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:17:55,060 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:17:55,060 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:17:55,060 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:17:55,273 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:17:56,273 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:18:00,061 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:18:02,620 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:18:02,620 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:18:05,788 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:18:06,907 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:18:06,908 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:18:06,908 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:18:06,908 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:18:07,276 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:18:08,276 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:18:10,909 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:18:13,954 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:18:15,955 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:18:17,620 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:18:17,620 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:18:18,746 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:18:18,746 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:18:18,746 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:18:18,747 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:18:19,278 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:18:20,279 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:18:21,747 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:18:26,748 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:18:30,616 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:18:30,617 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:18:30,617 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:18:30,617 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:18:31,281 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:18:32,281 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:18:32,618 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:18:32,620 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:18:32,620 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:18:37,840 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:18:42,474 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:18:42,475 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:18:42,475 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:18:42,475 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:18:43,284 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:18:43,476 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:18:43,947 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:18:44,284 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:18:47,620 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:18:47,620 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:18:48,841 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:18:53,842 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:18:58,287 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:18:59,223 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:19:02,620 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:19:02,621 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:19:04,821 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:19:08,929 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:19:08,929 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:19:08,929 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:19:09,069 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:19:09,289 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:19:10,070 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:19:10,290 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:19:13,947 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:19:15,948 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:19:17,620 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:19:17,621 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:19:20,737 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:19:20,737 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:19:20,737 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:19:20,738 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:19:21,292 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:19:21,738 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:19:22,292 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:19:26,739 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:19:31,739 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:19:32,550 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:19:32,551 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:19:32,551 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:19:32,552 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:19:32,620 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:19:32,621 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:19:33,295 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:19:34,295 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:19:36,839 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:19:41,840 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:19:43,948 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:19:44,389 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:19:44,390 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:19:44,390 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:19:44,390 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:19:45,298 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:19:46,298 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:19:47,391 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:19:47,621 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:19:47,621 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:19:52,856 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:19:56,237 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:19:56,237 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:19:56,237 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:19:56,238 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:19:56,300 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:19:58,238 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:19:58,301 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:20:02,621 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:20:02,621 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:20:03,760 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:20:08,138 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:20:08,139 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:20:08,139 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:20:08,140 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:20:08,303 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:20:09,140 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:20:10,304 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:20:13,948 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:20:14,949 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:20:19,950 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:20:20,438 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:20:20,438 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:20:25,610 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:20:29,210 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:20:29,211 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:20:29,211 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:20:29,211 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:20:29,308 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:20:30,308 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:20:31,211 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:20:35,438 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:20:35,438 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:20:36,579 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:20:40,998 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:20:40,998 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:20:40,999 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:20:41,000 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:20:41,310 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:20:42,000 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:20:42,311 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:20:43,949 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:20:47,950 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:20:50,438 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:20:50,438 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:20:52,834 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:20:52,834 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:20:52,835 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:20:52,835 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:20:53,313 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:20:53,835 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:20:54,313 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:20:58,836 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:21:03,837 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:21:04,660 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:21:04,661 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:21:04,661 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:21:04,661 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:21:05,316 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:21:05,438 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:21:05,439 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:21:06,316 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:21:09,609 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:21:13,949 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:21:14,950 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:21:16,526 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:21:16,526 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:21:16,527 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:21:16,527 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:21:17,319 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:21:18,319 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:21:20,438 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:21:20,438 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:21:20,439 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:21:25,605 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:21:28,374 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:21:28,374 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:21:28,375 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:21:28,375 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:21:29,321 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:21:30,322 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:21:31,375 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:21:35,438 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:21:35,438 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:21:36,664 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:21:40,230 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:21:40,230 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:21:40,231 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:21:40,231 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:21:40,324 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:21:42,231 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:21:42,325 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:21:43,950 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:21:47,951 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:21:50,438 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:21:50,439 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:21:52,062 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:21:52,062 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:21:52,063 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:21:52,063 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:21:52,327 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:21:53,063 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:21:54,327 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:21:58,064 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:22:03,064 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:22:03,842 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:22:03,842 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:22:03,843 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:22:03,843 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:22:04,330 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:22:05,438 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:22:05,439 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:22:06,330 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:22:08,677 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:22:13,678 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:22:13,950 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:22:15,742 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:22:15,742 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:22:15,743 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:22:15,743 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:22:16,332 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:22:18,333 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:22:18,743 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:22:20,439 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:22:20,439 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:22:24,672 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:22:27,574 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:22:27,574 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:22:27,575 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:22:27,575 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:22:28,335 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:22:30,336 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:22:30,575 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:22:35,439 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:22:35,439 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:22:35,666 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:22:39,438 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:22:39,438 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:22:39,439 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:22:39,439 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:22:40,338 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:22:40,338 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:22:41,439 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:22:43,951 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:22:46,952 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:22:50,439 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:22:50,439 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:22:51,294 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:22:51,294 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:22:51,295 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:22:51,295 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:22:51,340 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:22:52,295 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:22:52,341 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:22:57,296 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:23:02,297 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:23:03,092 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:23:03,092 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:23:03,093 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:23:03,093 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:23:03,343 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:23:04,343 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:23:05,439 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:23:05,439 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:23:07,651 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:23:12,652 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:23:13,955 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:23:14,986 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:23:14,986 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:23:14,987 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:23:14,987 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:23:15,346 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:23:16,346 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:23:17,987 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:23:20,439 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:23:20,439 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:23:23,621 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:23:26,830 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:23:26,830 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:23:26,831 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:23:26,831 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:23:27,348 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:23:28,349 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:23:28,831 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:23:33,832 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:23:35,439 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:23:35,439 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:23:38,694 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:23:38,694 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:23:38,695 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:23:38,695 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:23:39,351 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:23:39,695 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:23:40,351 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:23:43,952 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:23:44,953 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:23:49,953 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:23:50,439 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:23:50,440 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:23:50,550 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:23:50,610 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:23:50,610 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:23:50,611 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:23:51,354 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:23:52,354 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:23:55,612 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:24:00,612 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:24:02,362 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:24:02,363 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:24:02,363 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:24:02,363 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:24:03,356 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:24:04,357 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:24:05,439 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:24:05,440 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:24:06,572 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:24:11,572 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:24:13,952 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:24:14,270 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:24:14,270 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:24:14,271 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:24:14,271 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:24:14,359 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:24:16,360 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:24:17,271 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:24:20,439 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:24:20,440 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:24:22,568 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:24:26,118 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:24:26,118 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:24:26,119 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:24:26,119 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:24:26,362 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:24:28,120 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:24:28,362 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:24:33,121 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:24:35,439 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:24:35,440 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:24:37,962 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:24:37,963 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:24:37,963 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:24:37,963 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:24:38,365 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:24:38,963 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:24:40,365 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:24:43,953 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:24:44,953 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:24:49,773 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:24:49,774 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:24:49,774 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:24:49,775 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:24:50,367 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:24:50,439 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:24:50,440 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:24:50,440 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:24:52,368 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:24:55,588 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:25:00,589 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:25:01,682 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:25:01,683 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:25:01,683 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:25:01,684 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:25:02,370 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:25:02,370 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:25:05,440 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:25:05,440 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:25:05,673 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:25:10,674 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:25:13,538 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:25:13,539 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:25:13,539 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:25:13,539 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:25:13,953 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:25:14,373 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:25:14,373 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:25:15,954 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:25:20,440 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:25:20,440 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:25:21,620 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:25:25,418 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:25:25,418 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:25:25,419 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:25:25,419 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:25:26,375 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:25:26,376 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:25:27,420 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:25:32,420 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:25:35,440 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:25:35,440 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:25:37,270 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:25:37,271 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:25:37,271 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:25:37,273 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:25:37,378 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:25:38,273 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:25:38,378 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:25:43,274 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:25:43,954 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:25:48,955 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:25:49,095 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:25:49,095 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:25:49,096 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:25:49,096 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:25:49,381 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:25:50,381 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:25:50,440 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:25:50,440 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:25:54,653 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:25:59,654 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:26:00,978 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:26:00,978 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:26:00,979 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:26:00,979 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:26:01,383 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:26:02,384 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:26:04,980 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:26:05,440 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:26:05,441 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:26:10,585 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:26:12,918 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:26:12,919 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:26:12,919 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:26:12,920 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:26:13,386 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:26:13,954 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:26:14,386 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:26:15,955 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:26:20,441 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:26:20,441 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:26:21,660 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:26:24,778 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:26:24,779 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:26:24,779 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:26:24,779 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:26:25,389 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:26:26,389 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:26:26,779 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:26:31,780 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:26:35,440 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:26:35,441 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:26:36,642 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:26:36,642 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:26:36,643 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:26:36,643 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:26:37,392 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:26:37,643 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:26:38,392 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:26:42,644 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:26:43,955 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:26:47,956 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:26:48,443 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:26:48,444 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:26:48,444 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:26:48,445 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:26:49,395 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:26:50,395 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:26:50,440 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:26:50,441 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:26:53,603 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:26:58,604 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:27:00,334 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:27:00,334 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:27:00,335 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:27:00,335 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:27:00,397 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:27:02,398 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:27:04,336 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:27:05,441 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:27:05,441 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:27:09,678 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:27:12,166 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:27:12,166 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:27:12,167 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:27:12,167 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:27:12,400 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:27:13,955 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:27:14,401 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:27:14,956 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:27:19,957 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:27:20,441 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:27:20,441 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:27:24,030 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:27:24,031 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:27:24,031 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:27:24,031 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:27:24,403 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:27:25,031 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:27:26,404 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:27:30,032 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:27:35,033 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:27:35,441 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:27:35,441 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:27:35,890 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:27:35,891 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:27:35,891 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:27:35,891 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:27:36,406 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:27:36,406 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:27:40,892 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:27:43,956 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:27:45,957 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:27:47,694 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:27:47,695 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:27:47,695 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:27:47,695 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:27:48,409 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:27:48,409 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:27:50,441 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:27:50,441 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:27:51,661 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:27:56,662 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:27:59,578 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:27:59,578 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:27:59,579 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:27:59,579 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:28:00,412 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:28:00,413 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:28:02,580 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:28:05,441 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:28:05,441 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:28:07,630 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:28:11,414 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:28:11,414 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:28:11,415 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:28:11,415 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:28:11,415 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:28:12,416 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:28:13,415 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:28:13,956 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:28:18,957 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:28:20,441 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:28:20,441 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:28:23,274 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:28:23,274 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:28:23,275 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:28:23,275 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:28:23,418 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:28:24,275 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:28:24,419 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:28:29,276 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:28:34,277 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:28:35,071 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:28:35,072 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:28:35,072 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:28:35,072 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:28:35,421 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:28:35,441 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:28:35,442 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:28:36,422 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:28:39,610 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:28:43,957 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:28:44,957 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:28:46,958 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:28:46,958 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:28:46,959 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:28:46,959 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:28:47,424 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:28:48,425 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:28:49,960 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:28:50,441 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:28:50,442 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:28:55,661 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:28:58,810 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:28:58,810 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:28:58,811 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:28:58,811 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:28:59,427 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:29:00,428 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:29:00,811 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:29:05,441 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:29:05,442 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:29:06,634 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:29:10,670 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:29:10,670 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:29:10,671 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:29:10,671 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:29:11,430 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:29:11,671 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:29:12,431 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:29:13,957 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:29:16,958 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:29:20,441 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:29:20,442 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:29:22,476 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:29:22,518 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:29:22,518 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:29:22,519 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:29:22,519 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:29:23,433 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:29:24,434 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:29:27,520 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:29:32,520 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:29:34,308 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:29:34,309 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:29:34,309 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:29:34,309 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:29:34,436 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:29:35,442 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:29:35,442 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:29:36,437 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:29:37,640 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:29:42,641 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:29:43,958 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:29:46,190 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:29:46,190 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:29:46,191 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:29:46,191 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:29:46,439 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:29:48,191 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:29:48,440 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:29:50,442 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:29:50,442 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:29:53,677 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:29:58,050 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:29:58,050 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:29:58,051 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:29:58,051 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:29:58,442 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:29:58,442 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:29:59,051 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:30:04,052 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:30:05,442 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:30:05,442 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:30:09,623 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:30:09,890 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:30:09,890 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:30:09,891 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:30:09,891 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:30:10,445 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:30:10,445 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:30:13,958 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:30:14,959 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:30:19,960 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:30:20,442 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:30:20,442 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:30:21,762 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:30:21,762 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:30:21,763 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:30:21,763 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:30:22,449 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:30:22,449 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:30:25,763 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:30:30,764 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:30:33,580 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:30:33,581 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:30:33,581 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:30:33,581 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:30:34,452 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:30:34,452 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:30:35,442 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:30:35,442 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:30:36,661 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:30:41,662 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:30:43,959 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:30:45,454 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:30:45,455 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:30:45,455 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:30:45,455 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:30:45,455 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:30:46,455 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:30:47,456 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:30:50,442 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:30:50,442 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:30:52,662 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:30:57,314 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:30:57,314 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:30:57,315 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:30:57,315 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:30:57,458 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:30:58,315 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:30:58,458 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:31:03,316 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:31:05,442 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:31:05,443 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:31:08,633 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:31:09,162 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:31:09,162 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:31:09,163 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:31:09,163 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:31:09,461 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:31:10,461 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:31:13,959 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:31:13,960 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:31:18,961 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:31:20,442 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:31:20,443 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:31:20,994 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:31:20,994 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:31:20,995 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:31:20,995 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:31:21,463 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:31:22,464 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:31:23,995 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:31:28,996 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:31:32,790 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:31:32,790 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:31:32,791 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:31:32,791 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:31:33,466 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:31:34,466 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:31:34,791 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:31:35,442 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:31:35,443 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:31:40,596 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:31:43,960 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:31:44,662 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:31:44,662 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:31:44,663 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:31:44,663 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:31:45,469 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:31:45,663 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:31:46,469 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:31:50,442 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:31:50,443 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:31:51,594 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:31:56,486 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:31:56,486 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:31:56,487 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:31:56,487 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:31:57,471 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:31:57,487 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:31:58,472 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:32:02,488 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:32:05,443 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:32:05,443 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:32:07,586 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:32:08,306 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:32:08,306 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:32:08,307 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:32:08,307 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:32:08,474 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:32:10,474 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:32:13,307 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:32:13,960 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:32:18,961 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:32:20,090 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:32:20,091 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:32:20,091 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:32:20,091 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:32:20,443 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:32:20,443 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:32:20,476 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:32:20,476 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:32:24,592 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:32:29,593 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:32:31,962 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:32:31,962 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:32:31,963 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:32:31,963 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:32:32,479 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:32:32,479 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:32:34,963 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:32:35,443 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:32:35,443 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:32:40,627 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:32:43,794 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:32:43,794 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:32:43,795 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:32:43,795 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:32:43,961 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:32:44,482 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:32:44,482 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:32:45,962 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:32:50,443 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:32:50,444 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:32:51,615 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:32:55,662 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:32:55,666 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:32:55,666 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:32:55,666 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:32:56,485 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:32:56,485 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:32:56,667 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:33:01,667 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:33:05,443 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:33:05,444 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:33:07,465 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:33:07,506 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:33:07,506 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:33:07,507 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:33:07,507 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:33:08,488 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:33:08,488 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:33:12,508 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:33:13,962 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:33:17,962 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:33:19,298 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:33:19,299 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:33:19,299 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:33:19,299 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:33:19,491 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:33:20,443 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:33:20,444 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:33:20,491 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:33:23,615 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:33:28,616 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:33:31,178 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:33:31,178 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:33:31,179 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:33:31,179 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:33:31,494 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:33:33,494 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:33:34,179 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:33:35,444 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:33:35,444 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:33:39,579 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:33:43,034 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:33:43,035 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:33:43,035 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:33:43,035 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:33:43,497 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:33:43,962 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:33:44,963 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:33:45,497 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:33:49,963 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:33:50,444 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:33:50,444 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:33:54,878 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:33:54,878 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:33:54,879 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:33:54,879 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:33:55,500 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:33:55,879 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:33:57,500 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:34:00,880 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:34:05,444 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:34:05,444 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:34:06,645 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:34:06,742 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:34:06,742 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:34:06,743 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:34:06,743 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:34:07,502 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:34:09,503 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:34:11,744 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:34:13,963 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:34:16,963 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:34:18,554 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:34:18,555 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:34:18,555 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:34:18,555 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:34:19,506 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:34:20,444 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:34:20,444 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:34:21,506 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:34:22,620 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:34:27,621 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:34:30,450 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:34:30,450 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:34:30,451 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:34:30,451 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:34:30,508 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:34:33,451 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:34:33,509 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:34:35,444 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:34:35,444 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:34:38,631 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:34:42,370 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:34:42,371 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:34:42,371 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:34:42,371 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:34:42,512 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:34:43,512 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:34:43,963 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:34:43,964 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:34:48,964 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:34:50,444 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:34:50,445 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:34:54,166 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:34:54,206 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:34:54,206 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:34:54,207 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:34:54,207 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:34:54,515 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:34:55,515 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:34:59,208 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:35:04,208 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:35:05,444 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:35:05,445 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:35:06,046 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:35:06,047 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:35:06,047 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:35:06,047 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:35:06,518 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:35:07,518 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:35:10,048 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:35:13,964 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:35:15,964 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:35:17,835 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:35:17,835 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:35:17,836 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:35:17,836 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:35:18,521 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:35:19,521 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:35:20,444 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:35:20,445 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:35:21,594 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:35:26,595 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:35:29,746 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:35:29,747 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:35:29,747 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:35:29,747 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:35:30,524 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:35:31,524 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:35:31,748 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:35:35,445 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:35:35,445 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:35:37,671 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:35:41,578 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:35:41,579 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:35:41,579 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:35:41,579 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:35:42,527 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:35:43,528 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:35:43,580 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:35:43,964 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:35:48,965 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:35:50,445 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:35:50,445 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:35:53,406 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:35:53,407 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:35:53,407 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:35:53,407 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:35:53,530 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:35:54,408 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:35:55,531 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:35:59,408 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:36:04,409 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:36:05,182 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:36:05,182 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:36:05,183 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:36:05,183 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:36:05,445 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:36:05,445 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:36:05,533 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:36:07,534 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:36:09,600 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:36:13,965 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:36:14,965 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:36:17,058 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:36:17,059 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:36:17,059 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:36:17,059 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:36:17,537 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:36:19,537 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:36:20,060 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:36:20,445 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:36:20,445 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:36:25,606 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:36:28,926 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:36:28,927 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:36:28,927 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:36:28,927 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:36:29,540 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:36:30,928 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:36:31,540 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:36:35,445 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:36:35,445 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:36:36,588 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:36:40,770 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:36:40,771 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:36:40,771 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:36:40,771 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:36:41,543 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:36:41,771 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:36:43,544 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:36:43,965 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:36:46,966 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:36:50,445 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:36:50,446 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:36:52,582 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:36:52,626 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:36:52,626 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:36:52,627 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:36:52,627 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:36:53,546 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:36:55,547 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:36:57,628 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:37:02,628 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:37:04,435 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:37:04,436 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:37:04,436 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:37:04,436 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:37:04,549 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:37:05,445 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:37:05,445 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:37:05,550 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:37:08,595 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:37:13,596 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:37:13,966 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:37:16,314 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:37:16,315 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:37:16,315 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:37:16,315 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:37:16,552 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:37:17,553 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:37:19,316 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:37:20,445 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:37:20,446 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:37:24,577 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:37:28,142 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:37:28,143 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:37:28,143 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:37:28,143 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:37:28,555 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:37:29,556 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:37:30,144 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:37:35,144 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:37:35,446 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:37:35,446 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:37:39,978 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:37:39,979 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:37:39,979 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:37:39,979 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:37:40,558 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:37:40,979 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:37:41,559 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:37:43,966 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:37:46,967 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:37:50,446 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:37:50,446 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:37:51,822 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:37:51,823 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:37:51,823 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:37:51,823 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:37:52,561 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:37:52,823 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:37:53,562 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:37:57,824 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:38:02,825 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:38:03,612 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:38:03,612 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:38:03,613 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:38:03,613 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:38:04,564 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:38:05,446 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:38:05,446 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:38:05,565 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:38:08,650 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:38:13,651 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:38:13,966 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:38:15,486 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:38:15,487 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:38:15,487 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:38:15,487 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:38:15,567 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:38:17,568 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:38:19,488 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:38:20,446 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:38:20,446 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:38:24,622 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:38:27,370 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:38:27,371 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:38:27,371 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:38:27,371 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:38:27,570 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:38:29,571 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:38:30,372 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:38:35,372 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:38:35,446 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:38:35,446 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:38:39,214 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:38:39,215 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:38:39,215 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:38:39,215 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:38:39,573 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:38:41,216 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:38:41,574 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:38:43,967 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:38:46,968 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:38:50,446 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:38:50,446 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:38:51,050 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:38:51,051 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:38:51,051 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:38:51,051 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:38:51,577 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:38:52,051 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:38:53,577 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:38:57,052 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:39:02,053 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:39:05,447 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:39:05,447 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:39:07,581 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:39:07,589 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:39:12,590 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:39:13,968 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:39:17,079 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:39:17,080 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:39:17,080 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:39:17,188 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:39:17,584 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:39:18,190 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:39:19,585 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:39:20,446 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:39:20,446 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:39:23,581 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:39:28,581 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:39:28,863 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:39:28,864 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:39:28,864 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:39:28,864 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:39:29,587 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:39:31,587 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:39:33,865 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:39:35,446 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:39:35,447 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:39:39,626 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:39:40,687 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:39:40,687 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:39:40,688 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:39:40,688 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:39:41,590 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:39:41,590 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:39:43,968 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:39:44,969 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:39:49,969 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:39:50,446 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:39:50,447 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:39:52,566 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:39:52,567 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:39:52,567 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:39:52,567 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:39:52,593 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:39:53,594 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:39:55,568 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:40:00,568 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:40:04,367 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:40:04,368 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:40:04,368 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:40:04,368 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:40:04,597 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:40:05,447 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:40:05,447 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:40:05,597 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:40:05,600 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:40:10,601 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:40:13,969 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:40:15,969 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:40:16,218 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:40:16,219 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:40:16,219 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:40:16,219 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:40:16,600 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:40:17,601 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:40:20,447 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:40:20,447 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:40:21,676 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:40:26,677 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:40:28,081 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:40:28,081 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:40:28,082 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:40:28,082 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:40:28,604 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:40:29,604 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:40:32,083 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:40:35,447 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:40:35,448 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:40:37,666 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:40:39,920 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:40:39,921 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:40:39,921 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:40:39,921 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:40:40,607 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:40:41,607 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:40:42,922 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:40:43,969 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:40:47,970 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:40:50,447 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:40:50,447 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:40:51,774 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:40:51,775 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:40:51,775 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:40:51,775 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:40:52,610 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:40:53,611 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:40:53,775 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:40:58,776 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:41:03,614 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:41:03,615 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:41:03,615 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:41:03,615 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:41:04,613 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:41:04,615 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:41:05,447 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:41:05,447 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:41:05,614 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:41:09,620 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:41:13,970 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:41:14,970 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:41:15,459 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:41:15,460 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:41:15,460 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:41:15,460 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:41:15,616 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:41:17,617 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:41:20,447 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:41:20,447 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:41:20,447 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:41:25,654 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:41:27,307 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:41:27,308 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:41:27,308 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:41:27,308 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:41:27,619 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:41:29,620 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:41:31,309 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:41:35,448 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:41:35,448 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:41:36,661 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:41:39,162 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:41:39,162 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:41:39,162 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:41:39,162 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:41:39,623 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:41:41,623 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:41:42,163 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:41:43,970 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:41:47,971 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:41:50,448 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:41:50,448 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:41:51,016 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:41:51,017 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:41:51,017 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:41:51,017 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:41:51,626 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:41:53,018 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:41:53,626 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:41:58,018 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:42:02,880 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:42:02,880 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:42:02,881 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:42:02,881 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:42:03,628 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:42:03,881 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:42:05,448 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:42:05,448 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:42:05,629 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:42:09,607 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:42:13,971 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:42:14,725 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:42:14,725 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:42:14,725 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:42:14,726 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:42:14,726 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:42:15,631 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:42:15,631 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:42:19,727 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:42:20,448 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:42:20,448 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:42:25,681 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:42:26,561 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:42:26,562 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:42:26,562 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:42:26,562 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:42:26,634 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:42:27,634 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:42:31,563 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:42:35,448 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:42:35,448 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:42:36,631 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:42:38,373 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:42:38,373 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:42:38,373 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:42:38,374 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:42:38,637 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:42:39,637 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:42:42,374 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:42:43,971 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:42:47,972 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:42:50,215 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:42:50,219 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:42:50,219 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:42:50,219 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:42:50,448 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:42:50,448 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:42:50,639 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:42:51,640 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:42:53,593 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:42:58,594 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:43:02,044 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:43:02,045 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:43:02,045 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:43:02,045 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:43:02,642 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:43:03,643 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:43:04,046 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:43:05,448 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:43:05,448 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:43:09,639 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:43:13,888 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:43:13,889 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:43:13,889 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:43:13,889 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:43:13,972 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:43:14,645 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:43:14,972 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:43:15,645 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:43:19,973 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:43:20,448 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:43:20,448 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:43:25,626 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:43:25,732 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:43:25,733 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:43:25,733 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:43:25,733 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:43:26,648 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:43:27,648 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:43:30,734 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:43:35,448 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:43:35,449 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:43:36,683 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:43:37,573 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:43:37,574 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:43:37,574 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:43:37,574 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:43:37,651 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:43:39,651 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:43:42,575 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:43:43,972 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:43:47,973 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:43:49,407 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:43:49,407 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:43:49,408 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:43:49,408 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:43:49,654 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:43:50,449 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:43:50,449 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:43:51,655 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:43:53,671 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:43:58,671 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:44:01,264 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:44:01,264 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:44:01,265 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:44:01,265 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:44:01,657 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:44:03,658 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:44:04,265 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:44:05,449 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:44:05,449 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:44:09,671 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:44:13,108 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:44:13,109 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:44:13,109 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:44:13,109 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:44:13,661 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:44:13,973 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:44:14,973 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:44:15,661 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:44:19,974 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:44:20,449 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:44:20,449 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:44:24,954 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:44:24,955 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:44:24,955 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:44:24,955 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:44:25,664 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:44:25,664 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:44:25,956 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:44:30,956 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:44:35,957 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:44:38,281 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:44:38,282 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:44:41,501 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:44:43,973 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:44:46,974 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:44:49,542 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:44:49,542 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:44:49,543 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:44:49,543 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:44:49,670 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:44:51,670 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:44:52,544 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:44:53,281 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:44:53,281 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:44:58,431 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:45:01,358 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:45:01,358 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:45:01,359 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:45:01,359 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:45:01,673 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:45:03,673 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:45:04,359 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:45:08,281 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:45:08,281 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:45:09,501 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:45:13,190 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:45:13,190 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:45:13,191 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:45:13,191 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:45:13,676 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:45:13,974 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:45:14,974 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:45:15,676 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:45:19,975 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:45:23,281 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:45:23,281 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:45:24,985 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:45:25,026 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:45:25,026 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:45:25,027 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:45:25,027 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:45:25,679 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:45:25,679 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:45:30,028 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:45:35,028 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:45:36,836 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:45:36,836 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:45:36,837 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:45:36,837 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:45:37,682 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:45:37,682 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:45:38,281 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:45:38,281 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:45:40,414 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:45:43,974 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:45:45,975 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:45:48,714 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:45:48,714 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:45:48,715 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:45:48,715 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:45:49,684 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:45:49,685 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:45:51,716 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:45:53,281 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:45:53,282 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:45:57,478 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:46:00,558 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:46:00,558 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:46:00,559 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:46:00,559 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:46:00,687 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:46:01,688 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:46:02,559 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:46:07,560 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:46:08,281 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:46:08,282 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:46:12,406 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:46:12,406 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:46:12,407 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:46:12,407 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:46:12,690 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:46:13,407 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:46:13,691 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:46:13,975 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:46:18,976 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:46:23,281 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:46:23,282 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:46:24,208 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:46:24,250 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:46:24,250 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:46:24,251 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:46:24,251 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:46:24,693 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:46:25,694 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:46:29,252 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:46:34,252 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:46:36,093 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:46:36,094 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:46:36,094 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:46:36,094 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:46:36,696 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:46:37,696 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:46:38,281 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:46:38,282 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:46:39,441 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:46:43,975 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:46:44,976 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:46:47,998 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:46:47,998 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:46:47,999 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:46:47,999 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:46:48,699 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:46:49,699 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:46:49,999 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:46:53,282 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:46:53,282 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:46:55,510 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:46:59,842 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:46:59,842 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:46:59,843 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:46:59,843 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:47:00,702 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:47:00,843 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:47:01,703 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:47:05,844 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:47:08,282 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:47:08,282 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:47:11,419 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:47:11,690 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:47:11,691 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:47:11,691 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:47:11,691 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:47:11,705 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:47:13,706 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:47:13,976 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:47:16,977 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:47:21,977 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:47:23,282 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:47:23,282 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:47:23,558 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:47:23,558 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:47:23,559 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:47:23,559 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:47:23,708 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:47:25,709 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:47:27,560 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:47:32,560 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:47:35,349 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:47:35,349 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:47:35,350 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:47:35,350 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:47:35,711 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:47:37,712 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:47:38,282 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:47:38,282 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:47:38,282 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:47:43,463 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:47:43,976 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:47:47,266 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:47:47,270 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:47:47,271 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:47:47,271 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:47:47,714 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:47:47,715 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:47:49,271 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:47:53,282 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:47:53,282 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:47:54,414 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:47:59,110 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:47:59,110 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:47:59,111 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:47:59,111 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:47:59,717 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:47:59,717 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:48:00,111 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:48:05,112 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:48:08,282 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:48:08,283 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:48:10,414 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:48:10,934 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:48:10,934 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:48:10,935 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:48:10,935 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:48:11,720 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:48:11,720 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:48:13,977 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:48:15,978 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:48:20,978 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:48:22,746 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:48:22,747 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:48:22,747 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:48:22,747 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:48:23,282 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:48:23,283 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:48:23,723 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:48:23,723 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:48:26,527 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:48:31,528 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:48:34,658 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:48:34,659 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:48:34,659 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:48:34,659 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:48:34,726 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:48:35,726 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:48:36,659 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:48:38,283 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:48:38,283 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:48:42,494 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:48:43,977 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:48:46,502 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:48:46,502 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:48:46,503 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:48:46,503 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:48:46,728 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:48:47,503 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:48:47,729 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:48:52,504 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:48:53,283 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:48:53,283 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:48:58,313 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:48:58,354 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:48:58,354 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:48:58,355 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:48:58,355 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:48:58,731 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:48:59,732 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:49:03,356 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:49:08,283 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:49:08,283 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:49:08,442 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:49:10,198 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:49:10,198 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:49:10,199 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:49:10,199 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:49:10,734 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:49:11,735 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:49:13,978 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:49:13,978 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:49:18,979 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:49:22,004 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:49:22,004 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:49:22,004 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:49:22,005 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:49:22,737 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:49:23,283 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:49:23,283 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:49:23,738 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:49:24,493 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:49:29,494 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:49:33,882 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:49:33,882 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:49:33,883 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:49:33,883 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:49:34,740 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:49:34,883 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:49:35,741 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:49:38,283 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:49:38,283 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:49:40,487 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:49:43,978 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:49:45,667 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:49:45,710 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:49:45,710 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:49:45,711 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:49:45,711 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:49:45,743 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:49:47,744 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:49:50,712 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:49:53,283 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:49:53,283 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:49:56,516 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:49:57,513 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:49:57,514 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:49:57,514 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:49:57,514 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:49:57,746 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:49:59,747 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:50:02,515 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:50:07,516 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:50:08,283 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:50:08,283 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:50:09,414 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:50:09,415 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:50:09,415 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:50:09,415 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:50:09,749 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:50:09,749 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:50:13,416 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:50:13,979 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:50:18,980 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:50:21,247 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:50:21,247 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:50:21,248 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:50:21,248 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:50:21,752 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:50:21,752 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:50:23,283 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:50:23,283 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:50:24,496 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:50:29,497 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:50:33,118 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:50:33,118 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:50:33,119 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:50:33,119 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:50:33,755 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:50:33,755 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:50:35,120 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:50:38,283 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:50:38,284 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:50:40,486 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:50:43,980 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:50:44,990 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:50:44,990 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:50:44,991 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:50:44,991 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:50:45,759 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:50:45,759 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:50:45,991 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:50:50,992 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:50:53,283 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:50:53,284 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:50:56,521 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:50:56,826 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:50:56,826 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:50:56,827 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:50:56,827 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:50:57,762 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:50:57,762 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:51:01,828 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:51:06,828 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:51:08,284 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:51:08,284 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:51:08,658 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:51:08,658 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:51:08,659 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:51:08,659 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:51:08,765 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:51:09,765 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:51:12,660 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:51:13,980 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:51:17,981 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:51:20,469 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:51:20,470 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:51:20,470 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:51:20,470 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:51:20,767 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:51:22,768 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:51:23,284 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:51:23,284 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:51:23,284 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:51:28,412 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:51:32,382 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:51:32,383 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:51:32,383 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:51:32,383 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:51:32,770 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:51:34,384 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:51:34,771 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:51:38,284 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:51:38,284 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:51:39,514 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:51:43,981 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:51:44,234 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:51:44,234 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:51:44,235 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:51:44,235 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:51:44,773 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:51:45,235 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:51:46,773 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:51:50,236 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:51:53,284 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:51:53,284 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:51:55,519 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:51:56,094 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:51:56,094 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:51:56,095 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:51:56,095 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:51:56,776 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:51:58,776 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:52:01,096 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:52:06,097 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:52:07,899 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:52:07,900 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:52:07,900 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:52:07,900 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:52:08,285 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:52:08,285 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:52:08,779 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:52:10,779 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:52:11,467 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:52:13,981 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:52:16,982 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:52:19,801 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:52:19,802 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:52:19,802 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:52:19,803 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:52:20,782 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:52:22,782 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:52:22,803 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:52:23,284 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:52:23,285 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:52:28,429 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:52:31,702 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:52:31,703 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:52:31,703 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:52:31,703 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:52:31,785 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:52:33,704 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:52:34,785 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:52:38,285 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:52:38,285 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:52:39,459 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:52:43,598 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:52:43,598 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:52:43,599 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:52:43,599 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:52:43,788 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:52:43,986 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:52:44,788 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:52:44,987 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:52:49,987 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:52:53,285 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:52:53,285 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:52:55,435 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:52:55,478 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:52:55,478 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:52:55,479 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:52:55,479 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:52:55,791 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:52:56,791 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:53:00,480 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:53:05,481 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:53:07,309 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:53:07,310 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:53:07,310 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:53:07,311 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:53:07,794 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:53:08,285 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:53:08,285 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:53:08,794 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:53:11,480 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:53:13,982 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:53:16,983 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:53:19,230 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:53:19,230 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:53:19,231 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:53:19,232 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:53:19,797 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:53:20,797 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:53:22,232 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:53:23,285 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:53:23,285 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:53:27,450 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:53:31,070 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:53:31,071 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:53:31,071 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:53:31,071 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:53:31,800 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:53:32,800 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:53:33,072 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:53:38,072 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:53:38,285 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:53:38,285 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:53:42,914 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:53:42,915 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:53:42,915 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:53:42,915 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:53:43,803 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:53:43,915 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:53:43,983 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:53:44,804 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:53:48,984 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:53:53,285 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:53:53,286 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:53:54,429 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:53:54,762 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:53:54,762 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:53:54,763 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:53:54,764 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:53:54,806 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:53:56,807 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:53:59,765 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:54:04,765 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:54:06,628 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:54:06,629 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:54:06,629 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:54:06,629 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:54:06,809 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:54:08,285 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:54:08,286 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:54:08,810 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:54:10,471 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:54:13,983 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:54:15,984 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:54:18,538 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:54:18,539 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:54:18,539 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:54:18,539 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:54:18,813 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:54:20,813 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:54:21,540 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:54:23,285 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:54:23,286 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:54:27,525 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:54:30,398 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:54:30,399 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:54:30,399 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:54:30,400 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:54:30,816 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:54:32,816 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:54:33,400 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:54:38,285 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:54:38,286 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:54:38,501 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:54:42,250 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:54:42,251 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:54:42,251 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:54:42,251 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:54:42,819 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:54:43,984 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:54:43,984 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:54:44,820 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:54:48,985 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:54:53,285 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:54:53,286 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:54:54,056 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:54:54,098 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:54:54,098 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:54:54,099 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:54:54,099 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:54:54,822 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:54:56,823 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:54:59,100 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:55:04,100 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:55:05,913 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:55:05,914 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:55:05,914 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:55:05,915 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:55:06,825 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:55:08,286 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:55:08,286 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:55:08,826 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:55:09,476 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:55:13,984 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:55:14,985 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:55:17,818 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:55:17,819 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:55:17,819 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:55:17,819 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:55:17,828 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:55:20,819 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:55:20,829 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:55:23,286 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:55:23,286 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:55:26,503 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:55:29,702 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:55:29,703 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:55:29,703 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:55:29,703 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:55:29,832 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:55:30,832 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:55:31,703 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:55:36,704 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:55:38,286 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:55:38,286 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:55:41,562 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:55:41,562 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:55:41,563 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:55:41,564 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:55:41,835 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:55:42,564 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:55:42,835 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:55:43,985 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:55:47,986 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:55:52,986 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:55:53,286 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:55:53,286 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:55:53,406 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:55:53,522 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:55:53,523 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:55:53,523 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:55:53,838 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:55:54,838 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:55:58,524 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:56:03,525 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:56:05,210 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:56:05,211 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:56:05,211 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:56:05,211 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:56:05,841 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:56:06,841 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:56:08,286 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:56:08,287 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:56:09,431 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:56:13,985 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:56:14,986 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:56:17,074 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:56:17,074 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:56:17,075 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:56:17,075 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:56:17,843 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:56:18,844 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:56:20,075 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:56:23,286 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:56:23,287 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:56:25,452 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:56:28,922 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:56:28,922 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:56:28,923 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:56:28,923 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:56:29,846 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:56:30,847 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:56:30,923 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:56:35,924 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:56:38,286 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:56:38,287 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:56:40,778 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:56:40,778 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:56:40,779 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:56:40,779 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:56:40,849 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:56:41,779 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:56:42,850 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:56:43,986 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:56:46,987 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:56:51,987 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:56:52,583 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:56:52,583 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:56:52,583 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:56:52,584 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:56:52,852 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:56:53,286 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:56:53,287 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:56:54,853 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:56:57,521 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:57:02,522 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:57:04,462 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:57:04,462 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:57:04,463 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:57:04,463 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:57:04,856 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:57:06,856 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:57:08,287 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:57:08,287 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:57:08,287 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:57:13,439 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:57:13,986 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:57:16,306 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:57:16,306 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:57:16,307 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:57:16,307 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:57:16,859 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:57:18,860 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:57:19,307 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:57:23,287 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:57:23,287 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:57:24,440 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:57:28,198 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:57:28,198 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:57:28,199 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:57:28,199 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:57:28,862 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:57:30,199 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:57:30,863 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:57:35,200 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:57:38,287 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:57:38,287 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:57:40,050 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:57:40,050 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:57:40,051 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:57:40,051 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:57:40,865 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:57:41,051 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:57:42,866 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:57:43,991 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:57:46,991 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:57:51,866 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:57:51,867 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:57:51,867 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:57:51,867 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:57:51,868 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:57:52,868 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:57:52,868 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:57:53,287 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:57:53,287 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:57:58,510 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:58:03,511 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:58:03,754 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:58:03,754 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:58:03,755 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:58:03,755 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:58:03,871 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:58:04,871 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:58:08,287 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:58:08,287 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:58:09,480 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:58:13,987 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:58:14,988 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:58:15,598 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:58:15,598 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:58:15,599 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:58:15,600 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:58:15,874 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:58:16,874 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:58:20,600 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:58:23,287 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:58:23,288 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:58:26,478 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:58:27,474 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:58:27,474 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:58:27,475 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:58:27,475 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:58:27,877 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:58:28,878 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:58:32,476 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:58:37,476 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:58:38,287 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:58:38,288 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:58:39,318 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:58:39,318 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:58:39,319 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:58:39,319 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:58:39,880 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:58:40,881 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:58:43,319 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:58:43,988 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:58:48,988 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:58:51,156 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:58:51,157 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:58:51,157 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:58:51,158 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:58:51,883 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:58:52,884 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:58:53,287 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:58:53,288 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:58:54,432 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:58:59,433 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:59:03,066 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:59:03,066 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:59:03,067 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:59:03,067 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:59:03,886 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:59:04,887 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:59:05,067 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:59:08,287 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:59:08,288 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:59:10,477 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:59:13,988 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:59:15,989 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:59:18,890 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:59:21,387 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:59:23,288 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:59:23,288 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:59:26,419 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:59:29,098 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:59:29,099 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:59:29,099 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:59:29,236 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:59:29,892 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:59:30,893 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:59:32,237 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:59:37,238 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:59:38,288 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:59:38,288 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:59:40,906 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:59:40,906 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:59:40,907 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:59:40,907 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:59:41,895 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:59:42,896 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:59:42,907 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:59:43,989 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 22:59:47,990 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:59:52,687 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 22:59:52,687 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 22:59:52,688 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 22:59:52,688 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 22:59:52,898 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 22:59:53,288 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 22:59:53,288 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 22:59:53,288 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 22:59:54,899 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 22:59:58,454 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:00:03,455 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:00:04,562 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:00:04,562 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:00:04,563 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:00:04,564 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:00:04,902 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:00:06,902 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:00:08,288 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:00:08,288 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:00:09,444 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:00:13,989 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:00:14,990 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:00:16,406 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:00:16,406 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:00:16,407 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:00:16,407 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:00:16,905 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:00:18,906 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:00:20,407 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:00:23,288 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:00:23,288 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:00:25,514 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:00:28,246 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:00:28,246 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:00:28,247 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:00:28,247 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:00:28,908 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:00:30,909 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:00:31,247 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:00:36,248 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:00:38,288 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:00:38,288 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:00:40,102 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:00:40,102 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:00:40,103 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:00:40,104 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:00:40,912 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:00:40,912 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:00:42,104 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:00:43,990 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:00:47,991 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:00:51,908 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:00:51,908 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:00:51,909 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:00:51,909 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:00:51,915 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:00:52,915 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:00:53,288 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:00:53,289 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:00:53,289 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:00:58,412 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:01:03,413 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:01:03,806 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:01:03,806 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:01:03,807 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:01:03,807 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:01:03,917 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:01:04,918 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:01:08,288 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:01:08,289 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:01:08,481 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:01:13,482 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:01:13,990 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:01:15,674 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:01:15,674 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:01:15,675 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:01:15,675 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:01:15,920 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:01:16,921 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:01:18,675 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:01:23,288 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:01:23,289 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:01:24,520 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:01:27,510 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:01:27,510 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:01:27,511 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:01:27,511 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:01:27,923 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:01:28,924 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:01:30,511 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:01:35,512 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:01:38,289 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:01:38,289 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:01:39,350 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:01:39,350 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:01:39,351 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:01:39,351 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:01:39,926 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:01:40,926 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:01:41,351 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:01:43,991 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:01:46,992 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:01:51,161 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:01:51,161 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:01:51,162 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:01:51,162 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:01:51,929 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:01:52,162 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:01:52,929 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:01:53,289 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:01:53,289 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:01:57,517 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:02:02,518 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:02:03,058 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:02:03,058 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:02:03,059 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:02:03,059 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:02:03,932 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:02:04,932 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:02:08,060 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:02:08,289 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:02:08,289 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:02:13,425 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:02:13,991 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:02:14,898 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:02:14,898 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:02:14,899 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:02:14,899 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:02:14,934 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:02:16,935 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:02:18,899 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:02:23,289 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:02:23,289 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:02:24,487 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:02:26,758 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:02:26,758 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:02:26,759 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:02:26,759 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:02:26,937 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:02:28,938 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:02:29,759 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:02:34,760 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:02:38,289 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:02:38,290 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:02:38,610 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:02:38,610 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:02:38,611 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:02:38,611 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:02:38,940 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:02:40,611 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:02:40,941 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:02:43,996 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:02:45,996 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:02:50,402 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:02:50,402 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:02:50,403 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:02:50,403 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:02:50,943 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:02:51,403 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:02:52,944 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:02:53,289 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:02:53,289 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:02:56,435 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:03:01,435 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:03:02,294 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:03:02,294 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:03:02,295 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:03:02,295 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:03:02,946 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:03:02,946 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:03:07,296 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:03:08,289 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:03:08,290 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:03:12,505 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:03:13,992 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:03:14,122 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:03:14,122 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:03:14,123 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:03:14,124 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:03:14,949 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:03:14,949 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:03:18,124 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:03:23,125 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:03:23,289 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:03:23,290 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:03:25,962 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:03:25,962 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:03:25,963 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:03:25,963 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:03:26,952 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:03:26,952 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:03:28,963 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:03:33,964 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:03:37,771 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:03:37,771 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:03:37,771 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:03:37,772 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:03:37,955 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:03:38,289 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:03:38,290 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:03:38,955 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:03:39,471 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:03:43,993 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:03:44,993 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:03:49,646 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:03:49,646 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:03:49,647 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:03:49,648 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:03:49,957 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:03:50,648 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:03:50,958 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:03:53,290 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:03:53,290 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:03:56,483 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:04:01,494 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:04:01,494 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:04:01,495 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:04:01,495 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:04:01,495 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:04:01,960 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:04:02,961 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:04:06,496 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:04:08,290 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:04:08,290 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:04:12,463 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:04:13,330 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:04:13,330 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:04:13,331 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:04:13,331 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:04:13,963 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:04:13,993 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:04:14,963 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:04:17,994 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:04:22,995 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:04:23,290 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:04:23,290 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:04:25,178 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:04:25,178 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:04:25,179 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:04:25,180 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:04:25,965 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:04:26,966 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:04:28,180 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:04:33,181 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:04:36,957 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:04:36,957 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:04:36,958 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:04:36,958 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:04:36,968 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:04:38,290 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:04:38,290 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:04:38,290 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:04:38,969 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:04:43,437 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:04:43,994 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:04:48,783 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:04:48,826 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:04:48,826 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:04:48,827 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:04:48,827 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:04:48,971 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:04:50,972 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:04:53,290 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:04:53,290 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:04:54,434 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:04:59,435 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:05:00,678 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:05:00,678 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:05:00,679 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:05:00,680 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:05:00,974 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:05:02,975 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:05:04,680 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:05:08,290 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:05:08,291 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:05:10,458 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:05:12,502 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:05:12,502 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:05:12,503 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:05:12,503 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:05:12,977 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:05:13,995 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:05:14,977 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:05:15,995 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:05:20,996 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:05:23,290 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:05:23,291 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:05:24,370 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:05:24,370 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:05:24,371 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:05:24,371 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:05:24,980 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:05:24,980 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:05:26,371 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:05:31,372 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:05:36,199 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:05:36,200 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:05:36,200 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:05:36,201 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:05:36,982 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:05:36,982 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:05:37,201 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:05:38,290 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:05:38,291 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:05:42,460 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:05:43,995 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:05:47,996 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:05:48,082 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:05:48,082 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:05:48,083 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:05:48,083 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:05:48,985 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:05:48,985 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:05:53,084 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:05:53,290 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:05:53,291 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:05:58,484 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:05:59,930 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:05:59,931 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:05:59,931 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:05:59,931 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:05:59,988 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:06:00,988 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:06:03,932 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:06:08,291 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:06:08,291 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:06:09,427 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:06:11,782 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:06:11,783 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:06:11,783 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:06:11,784 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:06:11,990 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:06:12,991 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:06:13,996 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:06:14,996 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:06:19,997 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:06:23,291 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:06:23,291 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:06:23,622 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:06:23,622 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:06:23,623 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:06:23,623 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:06:23,993 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:06:24,994 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:06:25,623 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:06:30,624 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:06:35,438 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:06:35,438 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:06:35,439 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:06:35,439 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:06:35,996 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:06:36,439 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:06:36,997 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:06:38,291 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:06:38,291 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:06:42,436 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:06:43,996 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:06:47,326 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:06:47,327 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:06:47,327 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:06:47,327 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:06:48,000 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:06:48,327 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:06:49,000 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:06:53,291 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:06:53,291 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:06:53,455 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:06:58,455 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:06:59,194 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:06:59,194 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:06:59,195 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:06:59,195 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:07:00,003 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:07:01,003 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:07:04,196 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:07:08,291 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:07:08,291 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:07:09,449 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:07:11,042 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:07:11,043 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:07:11,043 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:07:11,043 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:07:12,006 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:07:13,006 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:07:13,997 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:07:14,997 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:07:19,998 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:07:22,838 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:07:22,839 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:07:22,839 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:07:22,839 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:07:23,009 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:07:23,291 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:07:23,292 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:07:25,009 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:07:25,508 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:07:30,509 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:07:34,714 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:07:34,718 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:07:34,718 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:07:34,718 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:07:35,012 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:07:35,719 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:07:37,012 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:07:38,291 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:07:38,292 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:07:41,487 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:07:43,997 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:07:46,517 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:07:46,558 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:07:46,558 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:07:46,559 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:07:46,559 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:07:47,015 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:07:47,015 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:07:51,560 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:07:53,291 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:07:53,292 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:07:57,501 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:07:58,398 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:07:58,399 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:07:58,399 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:07:58,399 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:07:59,018 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:07:59,018 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:08:03,400 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:08:08,292 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:08:08,292 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:08:08,492 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:08:10,226 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:08:10,227 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:08:10,227 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:08:10,227 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:08:11,020 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:08:11,021 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:08:13,998 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:08:13,998 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:08:18,999 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:08:22,035 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:08:22,036 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:08:22,036 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:08:22,036 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:08:23,023 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:08:23,023 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:08:23,292 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:08:23,292 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:08:24,467 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:08:29,467 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:08:33,926 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:08:33,926 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:08:33,927 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:08:33,927 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:08:34,026 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:08:34,927 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:08:35,026 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:08:39,928 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:08:43,998 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:08:44,999 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:08:48,771 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:08:48,771 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:08:50,989 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:08:55,990 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:08:57,554 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:08:57,554 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:08:57,555 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:08:57,555 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:08:58,031 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:08:59,031 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:09:01,556 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:09:03,771 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:09:03,771 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:09:06,978 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:09:09,311 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:09:09,312 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:09:09,312 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:09:09,312 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:09:10,034 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:09:12,034 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:09:12,313 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:09:13,999 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:09:18,000 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:09:18,771 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:09:18,771 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:09:21,178 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:09:21,179 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:09:21,179 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:09:21,179 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:09:22,036 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:09:23,180 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:09:24,037 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:09:28,181 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:09:33,008 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:09:33,008 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:09:33,009 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:09:33,009 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:09:33,039 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:09:33,771 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:09:33,771 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:09:33,986 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:09:36,040 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:09:38,986 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:09:43,987 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:09:43,999 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:09:44,873 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:09:44,874 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:09:44,874 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:09:44,874 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:09:45,042 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:09:48,043 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:09:48,771 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:09:48,771 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:09:49,904 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:09:54,904 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:09:56,721 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:09:56,722 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:09:56,722 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:09:56,722 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:09:57,045 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:09:58,045 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:10:00,723 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:10:03,771 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:10:03,771 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:10:05,941 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:10:08,576 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:10:08,576 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:10:08,576 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:10:08,577 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:10:09,048 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:10:10,048 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:10:11,577 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:10:14,000 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:10:17,000 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:10:18,771 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:10:18,772 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:10:20,406 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:10:20,407 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:10:20,407 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:10:20,407 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:10:21,051 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:10:22,051 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:10:22,408 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:10:27,408 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:10:32,243 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:10:32,243 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:10:32,244 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:10:32,244 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:10:33,054 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:10:33,244 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:10:33,771 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:10:33,772 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:10:34,054 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:10:38,981 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:10:43,982 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:10:44,000 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:10:44,066 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:10:44,066 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:10:44,066 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:10:44,067 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:10:45,057 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:10:46,057 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:10:48,771 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:10:48,772 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:10:48,990 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:10:53,991 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:10:55,925 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:10:55,925 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:10:55,925 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:10:55,926 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:10:56,060 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:10:58,061 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:10:59,926 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:11:03,771 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:11:03,772 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:11:04,978 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:11:07,745 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:11:07,745 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:11:07,745 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:11:07,745 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:11:08,064 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:11:10,064 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:11:10,746 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:11:14,001 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:11:16,001 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:11:18,772 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:11:18,772 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:11:19,577 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:11:19,577 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:11:19,577 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:11:19,577 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:11:20,067 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:11:21,578 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:11:22,068 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:11:26,579 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:11:31,435 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:11:31,436 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:11:31,436 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:11:31,436 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:11:32,070 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:11:32,436 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:11:33,772 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:11:33,772 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:11:34,071 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:11:37,941 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:11:42,941 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:11:43,287 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:11:43,288 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:11:43,288 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:11:43,288 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:11:44,001 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:11:44,073 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:11:46,074 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:11:48,002 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:11:48,772 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:11:48,772 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:11:53,980 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:11:55,122 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:11:55,123 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:11:55,123 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:11:55,123 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:11:56,076 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:11:58,077 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:11:59,124 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:12:03,772 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:12:03,772 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:12:04,929 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:12:06,971 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:12:06,971 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:12:06,972 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:12:06,972 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:12:07,079 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:12:09,972 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:12:10,080 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:12:14,001 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:12:15,002 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:12:18,772 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:12:18,773 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:12:18,814 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:12:18,978 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:12:18,978 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:12:18,979 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:12:19,083 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:12:20,979 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:12:22,084 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:12:25,980 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:12:30,648 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:12:30,649 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:12:30,649 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:12:30,649 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:12:31,086 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:12:31,650 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:12:32,087 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:12:33,772 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:12:33,773 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:12:36,997 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:12:41,998 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:12:42,488 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:12:42,505 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:12:42,505 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:12:42,505 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:12:43,090 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:12:44,002 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:12:44,090 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:12:47,003 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:12:48,772 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:12:48,773 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:12:52,006 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:12:54,326 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:12:54,326 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:12:54,326 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:12:54,326 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:12:55,093 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:12:56,093 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:12:57,327 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:13:02,328 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:13:03,772 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:13:03,773 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:13:06,148 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:13:06,148 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:13:06,149 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:13:06,149 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:13:07,096 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:13:08,097 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:13:08,149 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:13:13,150 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:13:14,003 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:13:17,982 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:13:17,983 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:13:17,983 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:13:17,983 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:13:18,099 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:13:18,772 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:13:18,773 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:13:18,956 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:13:20,100 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:13:23,956 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:13:28,957 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:13:29,800 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:13:29,801 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:13:29,801 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:13:29,801 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:13:30,102 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:13:32,103 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:13:33,773 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:13:33,773 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:13:34,004 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:13:39,004 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:13:41,629 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:13:41,630 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:13:41,630 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:13:41,630 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:13:42,106 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:13:44,003 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:13:44,106 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:13:45,004 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:13:48,773 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:13:48,773 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:13:50,996 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:13:53,491 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:13:53,491 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:13:53,491 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:13:53,492 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:13:54,108 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:13:56,109 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:13:56,492 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:14:01,493 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:14:03,773 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:14:03,773 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:14:05,338 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:14:05,339 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:14:05,339 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:14:05,339 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:14:06,111 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:14:07,340 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:14:08,112 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:14:12,341 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:14:14,004 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:14:17,177 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:14:17,178 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:14:17,178 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:14:17,178 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:14:18,114 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:14:18,179 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:14:18,773 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:14:18,773 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:14:20,115 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:14:23,992 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:14:28,992 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:14:29,029 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:14:29,030 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:14:29,030 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:14:29,030 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:14:29,117 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:14:30,117 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:14:33,773 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:14:33,773 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:14:34,003 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:14:39,004 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:14:40,890 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:14:40,891 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:14:40,891 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:14:40,891 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:14:41,120 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:14:42,120 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:14:44,004 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:14:44,005 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:14:48,773 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:14:48,773 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:14:49,936 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:14:52,731 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:14:52,732 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:14:52,732 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:14:52,732 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:14:53,123 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:14:54,124 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:14:55,733 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:15:00,733 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:15:03,773 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:15:03,774 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:15:04,559 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:15:04,560 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:15:04,560 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:15:04,560 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:15:05,127 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:15:06,127 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:15:06,561 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:15:11,562 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:15:14,005 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:15:16,391 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:15:16,391 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:15:16,391 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:15:16,392 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:15:17,130 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:15:17,392 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:15:18,130 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:15:18,773 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:15:18,774 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:15:23,011 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:15:28,012 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:15:28,300 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:15:28,301 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:15:28,301 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:15:28,301 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:15:29,133 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:15:30,134 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:15:33,302 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:15:33,773 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:15:33,774 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:15:38,993 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:15:40,136 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:15:40,136 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:15:40,136 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:15:40,137 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:15:40,137 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:15:42,137 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:15:44,005 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:15:44,006 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:15:48,773 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:15:48,774 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:15:49,927 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:15:51,980 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:15:51,980 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:15:51,981 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:15:51,981 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:15:52,140 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:15:54,140 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:15:54,982 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:15:59,982 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:16:03,774 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:16:03,774 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:16:03,808 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:16:04,000 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:16:04,000 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:16:04,000 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:16:04,143 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:16:05,001 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:16:06,144 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:16:10,001 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:16:14,006 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:16:15,006 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:16:15,632 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:16:15,633 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:16:15,633 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:16:15,633 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:16:16,147 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:16:18,147 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:16:18,774 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:16:18,774 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:16:20,990 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:16:25,991 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:16:27,479 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:16:27,480 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:16:27,480 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:16:27,480 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:16:28,150 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:16:30,151 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:16:31,481 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:16:33,774 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:16:33,774 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:16:36,997 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:16:39,337 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:16:39,337 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:16:39,338 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:16:39,338 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:16:40,154 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:16:42,154 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:16:42,339 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:16:44,006 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:16:48,007 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:16:48,774 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:16:48,774 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:16:51,181 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:16:51,182 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:16:51,182 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:16:51,182 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:16:52,157 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:16:52,157 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:16:53,183 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:16:58,183 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:17:03,041 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:17:03,041 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:17:03,041 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:17:03,042 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:17:03,160 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:17:03,774 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:17:03,774 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:17:03,938 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:17:04,161 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:17:08,938 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:17:13,939 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:17:14,007 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:17:14,873 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:17:14,874 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:17:14,874 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:17:14,874 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:17:15,164 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:17:16,164 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:17:18,774 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:17:18,775 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:17:19,907 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:17:24,907 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:17:26,711 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:17:26,712 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:17:26,712 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:17:26,713 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:17:27,167 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:17:28,167 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:17:30,713 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:17:33,774 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:17:33,775 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:17:35,960 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:17:38,558 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:17:38,584 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:17:38,584 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:17:38,585 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:17:39,170 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:17:40,170 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:17:41,585 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:17:44,007 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:17:47,008 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:17:48,774 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:17:48,775 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:17:50,387 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:17:50,388 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:17:50,388 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:17:50,388 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:17:51,173 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:17:52,173 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:17:52,389 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:17:57,390 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:18:02,221 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:18:02,222 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:18:02,222 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:18:02,222 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:18:03,176 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:18:03,223 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:18:03,775 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:18:03,775 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:18:04,176 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:18:08,977 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:18:13,978 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:18:14,008 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:18:14,061 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:18:14,061 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:18:14,061 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:18:14,061 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:18:14,179 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:18:16,180 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:18:18,775 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:18:18,775 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:18:19,927 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:18:24,928 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:18:25,900 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:18:25,900 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:18:25,901 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:18:25,901 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:18:26,183 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:18:28,183 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:18:30,902 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:18:33,775 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:18:33,775 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:18:35,955 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:18:37,750 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:18:37,750 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:18:37,751 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:18:37,751 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:18:38,186 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:18:40,187 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:18:41,752 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:18:44,008 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:18:47,009 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:18:48,775 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:18:48,775 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:18:49,576 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:18:49,576 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:18:49,577 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:18:49,577 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:18:50,190 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:18:52,191 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:18:52,578 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:18:57,578 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:19:01,426 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:19:01,427 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:19:01,427 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:19:01,427 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:19:02,193 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:19:03,428 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:19:03,775 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:19:03,775 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:19:04,194 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:19:08,911 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:19:13,261 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:19:13,262 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:19:13,262 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:19:13,262 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:19:14,009 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:19:14,009 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:19:14,197 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:19:14,197 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:19:18,775 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:19:18,775 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:19:19,993 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:19:24,994 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:19:30,202 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:19:30,387 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:19:33,775 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:19:33,776 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:19:35,983 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:19:40,077 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:19:40,077 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:19:40,078 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:19:40,204 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:19:40,324 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:19:41,325 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:19:42,205 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:19:44,010 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:19:47,010 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:19:48,775 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:19:48,776 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:19:51,851 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:19:51,851 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:19:51,851 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:19:51,852 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:19:52,208 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:19:52,852 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:19:54,208 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:19:57,853 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:20:02,854 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:20:03,663 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:20:03,664 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:20:03,664 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:20:03,664 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:20:03,775 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:20:03,776 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:20:04,211 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:20:06,212 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:20:07,981 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:20:12,982 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:20:14,010 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:20:15,495 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:20:15,495 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:20:15,496 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:20:15,496 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:20:16,214 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:20:18,215 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:20:18,497 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:20:18,775 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:20:18,776 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:20:23,985 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:20:27,338 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:20:27,338 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:20:27,339 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:20:27,339 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:20:28,218 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:20:28,218 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:20:29,339 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:20:33,776 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:20:33,776 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:20:34,965 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:20:39,197 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:20:39,198 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:20:39,198 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:20:39,198 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:20:39,221 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:20:40,199 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:20:40,221 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:20:44,011 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:20:46,011 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:20:48,776 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:20:48,776 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:20:51,040 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:20:51,041 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:20:51,041 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:20:51,041 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:20:51,041 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:20:51,224 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:20:52,224 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:20:56,042 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:21:01,043 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:21:02,873 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:21:02,874 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:21:02,874 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:21:02,874 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:21:03,227 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:21:03,776 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:21:03,776 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:21:04,227 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:21:07,001 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:21:12,002 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:21:14,011 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:21:14,716 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:21:14,716 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:21:14,717 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:21:14,717 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:21:15,230 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:21:16,230 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:21:17,718 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:21:18,776 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:21:18,776 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:21:22,935 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:21:26,566 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:21:26,566 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:21:26,566 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:21:26,567 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:21:27,233 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:21:28,233 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:21:28,567 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:21:33,568 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:21:33,776 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:21:33,776 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:21:38,399 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:21:38,400 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:21:38,400 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:21:38,400 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:21:39,236 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:21:39,401 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:21:40,237 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:21:44,012 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:21:45,012 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:21:48,776 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:21:48,776 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:21:50,242 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:21:50,242 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:21:50,243 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:21:50,243 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:21:50,243 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:21:51,239 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:21:52,240 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:21:55,244 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:22:00,245 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:22:02,064 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:22:02,065 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:22:02,065 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:22:02,065 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:22:02,242 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:22:03,776 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:22:03,776 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:22:04,243 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:22:05,917 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:22:10,918 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:22:13,898 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:22:13,899 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:22:13,899 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:22:13,899 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:22:14,012 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:22:14,245 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:22:16,013 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:22:16,246 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:22:18,776 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:22:18,777 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:22:21,918 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:22:25,750 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:22:25,754 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:22:25,755 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:22:25,755 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:22:26,248 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:22:27,756 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:22:28,249 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:22:32,756 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:22:33,776 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:22:33,777 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:22:37,597 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:22:37,597 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:22:37,598 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:22:37,598 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:22:38,251 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:22:38,598 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:22:40,252 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:22:43,599 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:22:44,012 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:22:48,776 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:22:48,777 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:22:48,961 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:22:49,422 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:22:49,422 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:22:49,422 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:22:49,423 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:22:50,254 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:22:50,254 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:22:54,424 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:22:59,424 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:23:01,275 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:23:01,276 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:23:01,276 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:23:01,276 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:23:02,257 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:23:02,257 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:23:03,777 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:23:03,777 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:23:04,898 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:23:09,899 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:23:13,088 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:23:13,088 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:23:13,089 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:23:13,089 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:23:13,260 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:23:14,013 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:23:14,260 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:23:15,014 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:23:18,777 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:23:18,777 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:23:21,007 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:23:24,906 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:23:24,907 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:23:24,907 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:23:24,907 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:23:25,263 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:23:26,263 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:23:26,908 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:23:31,909 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:23:33,777 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:23:33,777 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:23:36,729 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:23:36,730 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:23:36,730 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:23:36,730 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:23:37,266 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:23:37,731 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:23:38,267 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:23:42,731 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:23:44,013 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:23:48,014 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:23:48,564 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:23:48,564 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:23:48,565 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:23:48,565 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:23:48,777 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:23:48,777 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:23:49,269 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:23:50,270 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:23:53,914 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:23:58,915 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:24:00,394 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:24:00,395 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:24:00,395 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:24:00,395 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:24:01,272 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:24:02,273 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:24:03,777 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:24:03,777 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:24:03,978 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:24:08,979 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:24:12,219 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:24:12,220 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:24:12,220 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:24:12,220 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:24:12,275 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:24:14,014 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:24:14,014 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:24:14,276 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:24:18,777 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:24:18,777 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:24:19,992 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:24:24,062 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:24:24,062 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:24:24,063 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:24:24,063 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:24:24,279 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:24:25,063 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:24:26,279 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:24:30,064 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:24:33,777 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:24:33,777 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:24:35,899 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:24:35,899 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:24:35,899 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:24:35,900 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:24:35,900 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:24:36,281 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:24:38,282 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:24:40,901 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:24:44,015 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:24:46,015 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:24:47,737 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:24:47,737 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:24:47,737 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:24:47,738 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:24:48,285 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:24:48,285 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:24:48,777 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:24:48,778 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:24:51,992 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:24:56,993 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:24:59,563 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:24:59,564 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:24:59,564 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:24:59,564 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:25:00,288 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:25:00,288 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:25:02,565 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:25:03,777 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:25:03,778 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:25:07,996 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:25:11,417 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:25:11,418 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:25:11,418 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:25:11,418 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:25:12,291 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:25:12,291 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:25:13,419 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:25:14,015 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:25:18,778 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:25:18,778 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:25:18,971 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:25:23,221 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:25:23,222 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:25:23,222 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:25:23,222 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:25:23,294 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:25:24,222 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:25:24,294 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:25:29,223 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:25:33,778 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:25:33,778 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:25:35,008 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:25:35,048 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:25:35,048 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:25:35,049 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:25:35,049 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:25:35,297 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:25:36,297 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:25:40,050 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:25:44,016 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:25:46,016 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:25:46,884 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:25:46,885 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:25:46,885 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:25:46,885 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:25:47,300 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:25:48,300 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:25:48,778 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:25:48,778 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:25:51,939 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:25:56,940 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:25:58,727 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:25:58,727 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:25:58,728 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:25:58,728 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:25:59,303 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:26:00,303 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:26:02,729 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:26:03,778 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:26:03,778 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:26:07,965 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:26:10,598 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:26:10,598 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:26:10,599 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:26:10,599 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:26:11,306 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:26:12,307 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:26:13,600 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:26:14,016 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:26:18,778 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:26:18,778 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:26:18,965 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:26:22,440 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:26:22,440 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:26:22,441 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:26:22,441 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:26:23,309 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:26:24,310 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:26:24,441 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:26:29,442 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:26:33,778 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:26:33,778 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:26:34,255 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:26:34,256 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:26:34,256 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:26:34,256 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:26:34,312 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:26:35,256 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:26:36,313 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:26:40,257 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:26:44,017 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:26:46,017 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:26:46,071 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:26:46,071 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:26:46,071 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:26:46,072 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:26:46,315 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:26:48,316 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:26:48,778 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:26:48,778 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:26:52,009 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:26:57,009 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:26:57,891 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:26:57,891 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:26:57,891 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:26:57,892 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:26:58,318 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:26:59,319 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:27:02,892 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:27:03,779 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:27:03,779 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:27:08,067 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:27:09,721 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:27:09,722 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:27:09,722 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:27:09,722 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:27:10,321 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:27:11,321 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:27:13,723 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:27:14,020 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:27:18,778 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:27:18,779 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:27:18,942 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:27:21,576 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:27:21,576 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:27:21,576 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:27:21,577 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:27:22,324 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:27:23,324 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:27:24,577 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:27:29,578 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:27:33,400 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:27:33,401 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:27:33,401 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:27:33,401 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:27:33,779 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:27:33,779 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:27:34,327 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:27:34,965 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:27:35,327 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:27:39,966 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:27:44,018 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:27:45,018 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:27:45,237 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:27:45,237 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:27:45,237 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:27:45,238 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:27:45,330 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:27:47,330 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:27:48,779 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:27:48,779 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:27:50,929 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:27:55,930 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:27:57,067 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:27:57,068 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:27:57,068 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:27:57,068 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:27:57,333 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:27:59,334 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:28:01,069 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:28:03,779 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:28:03,779 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:28:06,932 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:28:08,899 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:28:08,900 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:28:08,900 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:28:08,900 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:28:09,336 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:28:11,337 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:28:12,901 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:28:14,018 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:28:18,019 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:28:18,779 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:28:18,779 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:28:20,749 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:28:20,749 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:28:20,749 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:28:20,750 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:28:21,340 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:28:23,340 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:28:23,751 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:28:28,752 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:28:32,592 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:28:32,593 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:28:32,593 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:28:32,593 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:28:33,343 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:28:33,779 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:28:33,779 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:28:33,917 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:28:35,344 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:28:38,917 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:28:43,918 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:28:44,019 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:28:44,414 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:28:44,414 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:28:44,414 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:28:44,415 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:28:45,347 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:28:47,347 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:28:48,779 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:28:48,779 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:28:49,020 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:28:54,021 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:28:56,247 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:28:56,248 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:28:56,248 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:28:56,249 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:28:56,350 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:28:59,250 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:28:59,351 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:29:03,779 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:29:03,779 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:29:04,986 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:29:08,085 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:29:08,085 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:29:08,085 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:29:08,086 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:29:08,353 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:29:10,086 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:29:11,354 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:29:14,019 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:29:16,020 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:29:18,779 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:29:18,779 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:29:19,927 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:29:19,927 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:29:19,928 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:29:19,928 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:29:20,357 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:29:21,357 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:29:21,928 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:29:26,929 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:29:31,763 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:29:31,764 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:29:31,764 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:29:31,765 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:29:32,360 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:29:32,766 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:29:33,361 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:29:33,779 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:29:33,779 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:29:37,910 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:29:42,910 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:29:43,601 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:29:43,601 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:29:43,601 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:29:43,602 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:29:44,020 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:29:44,364 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:29:45,364 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:29:48,020 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:29:48,780 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:29:48,780 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:29:53,999 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:29:55,445 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:29:55,446 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:29:55,446 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:29:55,446 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:29:56,367 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:29:57,367 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:29:59,447 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:30:03,779 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:30:03,780 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:30:04,997 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:30:07,304 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:30:07,304 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:30:07,304 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:30:07,305 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:30:07,370 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:30:09,371 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:30:10,306 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:30:14,020 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:30:16,021 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:30:18,780 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:30:18,780 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:30:19,148 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:30:19,149 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:30:19,149 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:30:19,149 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:30:19,373 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:30:21,150 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:30:21,374 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:30:26,150 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:30:30,984 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:30:30,985 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:30:30,985 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:30:30,985 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:30:31,376 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:30:31,985 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:30:33,377 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:30:33,780 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:30:33,780 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:30:37,903 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:30:42,837 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:30:42,838 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:30:42,838 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:30:42,839 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:30:43,380 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:30:43,839 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:30:44,021 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:30:45,380 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:30:48,780 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:30:48,780 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:30:48,899 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:30:53,900 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:30:54,678 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:30:54,678 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:30:54,678 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:30:54,678 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:30:55,383 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:30:57,384 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:30:59,679 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:31:03,780 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:31:03,780 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:31:04,907 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:31:06,512 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:31:06,513 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:31:06,513 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:31:06,513 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:31:07,386 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:31:09,387 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:31:10,514 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:31:14,021 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:31:16,022 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:31:18,340 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:31:18,341 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:31:18,341 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:31:18,341 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:31:18,389 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:31:18,780 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:31:18,780 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:31:21,390 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:31:21,921 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:31:26,922 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:31:30,172 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:31:30,173 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:31:30,173 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:31:30,173 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:31:30,393 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:31:31,393 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:31:32,174 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:31:33,780 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:31:33,781 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:31:37,969 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:31:42,015 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:31:42,015 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:31:42,015 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:31:42,016 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:31:42,396 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:31:43,016 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:31:43,396 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:31:44,022 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:31:48,022 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:31:48,780 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:31:48,781 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:31:53,842 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:31:53,842 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:31:53,842 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:31:53,842 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:31:53,843 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:31:54,399 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:31:55,399 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:31:58,844 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:32:03,781 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:32:03,781 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:32:04,001 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:32:05,683 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:32:05,684 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:32:05,684 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:32:05,684 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:32:06,402 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:32:07,402 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:32:09,685 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:32:14,029 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:32:15,030 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:32:17,511 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:32:17,511 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:32:17,512 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:32:17,512 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:32:18,405 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:32:18,781 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:32:18,781 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:32:19,405 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:32:20,908 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:32:25,909 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:32:29,341 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:32:29,341 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:32:29,342 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:32:29,342 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:32:29,408 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:32:31,342 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:32:31,408 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:32:33,781 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:32:33,781 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:32:37,015 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:32:41,168 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:32:41,168 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:32:41,169 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:32:41,169 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:32:41,410 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:32:42,169 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:32:43,411 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:32:44,023 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:32:48,023 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:32:48,781 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:32:48,781 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:32:53,024 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:32:53,025 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:32:53,025 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:32:53,025 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:32:53,026 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:32:53,413 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:32:55,414 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:32:58,027 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:33:03,027 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:33:03,781 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:33:03,781 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:33:04,861 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:33:04,862 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:33:04,862 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:33:04,862 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:33:05,416 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:33:07,417 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:33:08,863 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:33:13,864 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:33:14,023 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:33:18,781 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:33:18,782 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:33:18,967 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:33:23,968 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:33:28,140 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:33:28,141 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:33:28,141 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:33:28,142 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:33:28,422 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:33:29,142 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:33:29,422 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:33:33,781 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:33:33,782 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:33:34,993 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:33:39,916 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:33:39,917 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:33:39,917 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:33:39,917 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:33:40,425 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:33:40,917 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:33:41,425 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:33:44,024 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:33:46,024 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:33:48,781 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:33:48,782 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:33:51,743 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:33:51,743 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:33:51,743 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:33:51,743 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:33:51,744 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:33:52,427 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:33:53,428 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:33:56,744 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:34:01,745 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:34:03,574 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:34:03,575 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:34:03,575 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:34:03,576 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:34:03,782 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:34:03,782 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:34:04,430 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:34:05,430 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:34:06,908 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:34:11,909 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:34:14,024 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:34:15,404 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:34:15,404 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:34:15,404 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:34:15,405 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:34:15,432 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:34:17,405 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:34:17,433 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:34:18,782 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:34:18,782 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:34:22,970 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:34:27,246 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:34:27,247 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:34:27,247 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:34:27,247 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:34:27,435 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:34:28,248 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:34:29,436 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:34:33,248 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:34:33,782 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:34:33,782 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:34:39,002 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:34:39,095 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:34:39,096 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:34:39,096 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:34:39,097 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:34:39,438 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:34:41,439 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:34:44,025 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:34:44,025 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:34:48,782 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:34:48,782 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:34:49,950 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:34:50,953 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:34:50,954 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:34:50,954 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:34:50,954 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:34:51,442 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:34:53,442 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:34:54,955 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:34:59,955 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:35:02,809 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:35:02,809 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:35:02,809 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:35:02,810 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:35:03,445 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:35:03,782 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:35:03,782 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:35:05,446 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:35:05,821 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:35:10,822 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:35:14,025 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:35:14,677 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:35:14,678 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:35:14,678 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:35:14,678 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:35:15,448 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:35:16,679 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:35:17,449 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:35:18,782 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:35:18,783 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:35:21,917 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:35:26,509 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:35:26,510 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:35:26,510 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:35:26,510 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:35:27,452 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:35:27,510 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:35:29,452 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:35:32,511 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:35:33,782 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:35:33,783 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:35:37,961 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:35:38,355 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:35:38,355 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:35:38,356 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:35:38,356 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:35:38,454 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:35:39,455 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:35:43,357 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:35:44,026 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:35:48,782 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:35:48,783 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:35:48,938 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:35:50,199 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:35:50,199 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:35:50,199 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:35:50,200 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:35:50,458 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:35:51,458 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:35:54,200 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:35:59,201 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:36:02,037 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:36:02,038 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:36:02,038 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:36:02,038 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:36:02,460 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:36:03,460 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:36:03,783 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:36:03,783 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:36:04,948 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:36:09,949 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:36:13,879 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:36:13,879 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:36:13,879 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:36:13,880 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:36:14,026 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:36:14,463 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:36:15,027 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:36:15,463 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:36:18,783 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:36:18,783 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:36:20,971 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:36:25,731 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:36:25,731 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:36:25,732 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:36:25,732 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:36:26,466 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:36:26,732 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:36:27,466 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:36:31,733 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:36:33,783 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:36:33,783 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:36:36,945 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:36:37,610 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:36:37,611 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:36:37,611 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:36:37,611 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:36:38,468 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:36:39,469 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:36:42,612 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:36:44,027 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:36:48,028 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:36:48,783 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:36:48,783 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:36:49,460 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:36:49,461 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:36:49,461 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:36:49,461 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:36:49,471 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:36:51,472 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:36:53,462 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:36:58,463 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:37:01,297 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:37:01,298 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:37:01,298 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:37:01,298 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:37:01,474 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:37:03,474 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:37:03,783 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:37:03,783 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:37:03,924 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:37:08,925 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:37:13,143 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:37:13,143 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:37:13,143 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:37:13,144 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:37:13,477 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:37:14,031 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:37:14,032 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:37:15,477 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:37:18,783 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:37:18,784 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:37:20,000 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:37:24,980 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:37:24,981 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:37:24,981 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:37:24,981 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:37:25,479 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:37:25,982 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:37:27,480 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:37:30,983 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:37:33,783 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:37:33,784 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:37:36,810 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:37:36,811 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:37:36,811 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:37:36,811 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:37:36,811 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:37:37,482 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:37:39,483 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:37:41,812 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:37:44,028 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:37:47,028 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:37:48,659 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:37:48,659 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:37:48,660 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:37:48,660 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:37:48,783 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:37:48,784 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:37:49,485 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:37:51,486 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:37:53,018 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:37:58,019 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:38:00,500 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:38:00,500 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:38:00,500 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:38:00,501 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:38:01,488 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:38:01,488 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:38:03,501 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:38:03,784 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:38:03,784 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:38:08,937 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:38:12,337 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:38:12,337 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:38:12,338 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:38:12,338 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:38:12,491 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:38:13,491 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:38:14,028 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:38:14,029 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:38:18,784 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:38:18,784 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:38:19,946 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:38:24,195 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:38:24,195 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:38:24,196 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:38:24,197 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:38:24,493 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:38:25,197 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:38:25,494 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:38:30,198 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:38:33,784 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:38:33,784 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:38:35,999 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:38:36,058 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:38:36,059 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:38:36,059 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:38:36,059 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:38:36,496 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:38:37,497 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:38:41,060 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:38:44,029 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:38:47,030 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:38:47,912 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:38:47,913 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:38:47,913 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:38:47,913 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:38:48,499 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:38:48,784 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:38:48,784 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:38:49,499 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:38:52,994 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:38:57,994 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:38:59,759 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:38:59,759 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:38:59,760 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:38:59,761 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:39:00,502 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:39:01,502 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:39:03,761 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:39:03,784 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:39:03,784 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:39:09,002 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:39:11,610 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:39:11,610 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:39:11,611 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:39:11,611 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:39:12,505 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:39:13,505 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:39:14,029 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:39:14,030 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:39:18,784 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:39:18,785 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:39:19,918 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:39:23,465 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: partial_history 2024-05-25 23:39:23,465 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:39:23,465 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:39:23,466 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:39:23,508 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:39:25,466 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:39:25,508 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:39:30,467 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:39:35,468 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:39:37,817 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: stop_status 2024-05-25 23:39:37,818 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: stop_status 2024-05-25 23:39:38,634 DEBUG SenderThread:43320 [sender.py:send():378] send: telemetry 2024-05-25 23:39:38,634 DEBUG SenderThread:43320 [sender.py:send():378] send: exit 2024-05-25 23:39:38,634 INFO SenderThread:43320 [sender.py:send_exit():585] handling exit code: 0 2024-05-25 23:39:38,635 INFO SenderThread:43320 [sender.py:send_exit():587] handling runtime: 6774 2024-05-25 23:39:38,636 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:39:38,636 INFO SenderThread:43320 [sender.py:send_exit():593] send defer 2024-05-25 23:39:38,636 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: defer 2024-05-25 23:39:38,636 INFO HandlerThread:43320 [handler.py:handle_request_defer():184] handle defer: 0 2024-05-25 23:39:38,636 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: defer 2024-05-25 23:39:38,636 INFO SenderThread:43320 [sender.py:send_request_defer():609] handle sender defer: 0 2024-05-25 23:39:38,636 INFO SenderThread:43320 [sender.py:transition_state():613] send defer: 1 2024-05-25 23:39:38,636 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: defer 2024-05-25 23:39:38,636 INFO HandlerThread:43320 [handler.py:handle_request_defer():184] handle defer: 1 2024-05-25 23:39:38,636 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: defer 2024-05-25 23:39:38,636 INFO SenderThread:43320 [sender.py:send_request_defer():609] handle sender defer: 1 2024-05-25 23:39:38,636 INFO SenderThread:43320 [sender.py:transition_state():613] send defer: 2 2024-05-25 23:39:38,636 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: defer 2024-05-25 23:39:38,636 INFO HandlerThread:43320 [handler.py:handle_request_defer():184] handle defer: 2 2024-05-25 23:39:38,636 INFO HandlerThread:43320 [system_monitor.py:finish():203] Stopping system monitor 2024-05-25 23:39:38,636 DEBUG SystemMonitor:43320 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-05-25 23:39:38,637 DEBUG SystemMonitor:43320 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-05-25 23:39:38,637 INFO HandlerThread:43320 [interfaces.py:finish():200] Joined cpu monitor 2024-05-25 23:39:38,637 INFO HandlerThread:43320 [interfaces.py:finish():200] Joined disk monitor 2024-05-25 23:39:38,660 INFO HandlerThread:43320 [interfaces.py:finish():200] Joined gpu monitor 2024-05-25 23:39:38,660 INFO HandlerThread:43320 [interfaces.py:finish():200] Joined memory monitor 2024-05-25 23:39:38,660 INFO HandlerThread:43320 [interfaces.py:finish():200] Joined network monitor 2024-05-25 23:39:38,660 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: defer 2024-05-25 23:39:38,660 INFO SenderThread:43320 [sender.py:send_request_defer():609] handle sender defer: 2 2024-05-25 23:39:38,660 INFO SenderThread:43320 [sender.py:transition_state():613] send defer: 3 2024-05-25 23:39:38,660 DEBUG SenderThread:43320 [sender.py:send():378] send: stats 2024-05-25 23:39:38,660 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: defer 2024-05-25 23:39:38,660 INFO HandlerThread:43320 [handler.py:handle_request_defer():184] handle defer: 3 2024-05-25 23:39:38,661 DEBUG SenderThread:43320 [sender.py:send():378] send: history 2024-05-25 23:39:38,661 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: summary_record 2024-05-25 23:39:38,661 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:39:38,661 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: defer 2024-05-25 23:39:38,661 INFO SenderThread:43320 [sender.py:send_request_defer():609] handle sender defer: 3 2024-05-25 23:39:38,661 INFO SenderThread:43320 [sender.py:transition_state():613] send defer: 4 2024-05-25 23:39:38,661 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: defer 2024-05-25 23:39:38,661 INFO HandlerThread:43320 [handler.py:handle_request_defer():184] handle defer: 4 2024-05-25 23:39:38,661 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: defer 2024-05-25 23:39:38,661 INFO SenderThread:43320 [sender.py:send_request_defer():609] handle sender defer: 4 2024-05-25 23:39:38,661 INFO SenderThread:43320 [sender.py:transition_state():613] send defer: 5 2024-05-25 23:39:38,661 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: defer 2024-05-25 23:39:38,661 INFO HandlerThread:43320 [handler.py:handle_request_defer():184] handle defer: 5 2024-05-25 23:39:38,661 DEBUG SenderThread:43320 [sender.py:send():378] send: summary 2024-05-25 23:39:38,662 INFO SenderThread:43320 [sender.py:_save_file():1389] saving file wandb-summary.json with policy end 2024-05-25 23:39:38,662 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: defer 2024-05-25 23:39:38,662 INFO SenderThread:43320 [sender.py:send_request_defer():609] handle sender defer: 5 2024-05-25 23:39:38,662 INFO SenderThread:43320 [sender.py:transition_state():613] send defer: 6 2024-05-25 23:39:38,662 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: defer 2024-05-25 23:39:38,662 INFO HandlerThread:43320 [handler.py:handle_request_defer():184] handle defer: 6 2024-05-25 23:39:38,662 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: defer 2024-05-25 23:39:38,662 INFO SenderThread:43320 [sender.py:send_request_defer():609] handle sender defer: 6 2024-05-25 23:39:38,664 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: status_report 2024-05-25 23:39:38,813 INFO SenderThread:43320 [sender.py:transition_state():613] send defer: 7 2024-05-25 23:39:38,813 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: defer 2024-05-25 23:39:38,813 INFO HandlerThread:43320 [handler.py:handle_request_defer():184] handle defer: 7 2024-05-25 23:39:38,813 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: defer 2024-05-25 23:39:38,813 INFO SenderThread:43320 [sender.py:send_request_defer():609] handle sender defer: 7 2024-05-25 23:39:39,511 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:39:39,512 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/config.yaml 2024-05-25 23:39:39,512 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:39:39,606 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: poll_exit 2024-05-25 23:39:41,261 INFO SenderThread:43320 [sender.py:transition_state():613] send defer: 8 2024-05-25 23:39:41,262 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: poll_exit 2024-05-25 23:39:41,262 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: defer 2024-05-25 23:39:41,262 INFO HandlerThread:43320 [handler.py:handle_request_defer():184] handle defer: 8 2024-05-25 23:39:41,262 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: defer 2024-05-25 23:39:41,262 INFO SenderThread:43320 [sender.py:send_request_defer():609] handle sender defer: 8 2024-05-25 23:39:41,262 INFO SenderThread:43320 [job_builder.py:build():432] Attempting to build job artifact 2024-05-25 23:39:41,263 INFO SenderThread:43320 [job_builder.py:_get_source_type():565] is repo sourced job 2024-05-25 23:39:41,263 WARNING SenderThread:43320 [job_builder.py:_log_if_verbose():267] No program path found, not creating job artifact. See https://docs.wandb.ai/guides/launch/create-job 2024-05-25 23:39:41,263 INFO SenderThread:43320 [sender.py:transition_state():613] send defer: 9 2024-05-25 23:39:41,263 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: defer 2024-05-25 23:39:41,263 INFO HandlerThread:43320 [handler.py:handle_request_defer():184] handle defer: 9 2024-05-25 23:39:41,263 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: defer 2024-05-25 23:39:41,263 INFO SenderThread:43320 [sender.py:send_request_defer():609] handle sender defer: 9 2024-05-25 23:39:41,263 INFO SenderThread:43320 [dir_watcher.py:finish():358] shutting down directory watcher 2024-05-25 23:39:41,512 INFO Thread-12 :43320 [dir_watcher.py:_on_file_modified():288] file/dir modified: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:39:41,513 INFO SenderThread:43320 [dir_watcher.py:finish():388] scan: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files 2024-05-25 23:39:41,513 INFO SenderThread:43320 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/conda-environment.yaml conda-environment.yaml 2024-05-25 23:39:41,513 INFO SenderThread:43320 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log output.log 2024-05-25 23:39:41,513 INFO SenderThread:43320 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-metadata.json wandb-metadata.json 2024-05-25 23:39:41,513 INFO SenderThread:43320 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/config.yaml config.yaml 2024-05-25 23:39:41,513 INFO SenderThread:43320 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/requirements.txt requirements.txt 2024-05-25 23:39:41,513 INFO SenderThread:43320 [dir_watcher.py:finish():402] scan save: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json wandb-summary.json 2024-05-25 23:39:41,513 INFO SenderThread:43320 [sender.py:transition_state():613] send defer: 10 2024-05-25 23:39:41,515 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: defer 2024-05-25 23:39:41,515 INFO HandlerThread:43320 [handler.py:handle_request_defer():184] handle defer: 10 2024-05-25 23:39:41,516 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: defer 2024-05-25 23:39:41,516 INFO SenderThread:43320 [sender.py:send_request_defer():609] handle sender defer: 10 2024-05-25 23:39:41,517 INFO SenderThread:43320 [file_pusher.py:finish():169] shutting down file pusher 2024-05-25 23:39:41,607 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: poll_exit 2024-05-25 23:39:41,607 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: poll_exit 2024-05-25 23:39:41,861 INFO wandb-upload_1:43320 [upload_job.py:push():130] Uploaded file /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/conda-environment.yaml 2024-05-25 23:39:41,879 INFO wandb-upload_4:43320 [upload_job.py:push():130] Uploaded file /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/wandb-summary.json 2024-05-25 23:39:41,886 INFO wandb-upload_2:43320 [upload_job.py:push():130] Uploaded file /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/config.yaml 2024-05-25 23:39:42,088 INFO wandb-upload_3:43320 [upload_job.py:push():130] Uploaded file /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/requirements.txt 2024-05-25 23:39:42,129 INFO wandb-upload_0:43320 [upload_job.py:push():130] Uploaded file /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/files/output.log 2024-05-25 23:39:42,330 INFO Thread-11 (_thread_body):43320 [sender.py:transition_state():613] send defer: 11 2024-05-25 23:39:42,330 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: defer 2024-05-25 23:39:42,330 INFO HandlerThread:43320 [handler.py:handle_request_defer():184] handle defer: 11 2024-05-25 23:39:42,331 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: defer 2024-05-25 23:39:42,331 INFO SenderThread:43320 [sender.py:send_request_defer():609] handle sender defer: 11 2024-05-25 23:39:42,331 INFO SenderThread:43320 [file_pusher.py:join():175] waiting for file pusher 2024-05-25 23:39:42,331 INFO SenderThread:43320 [sender.py:transition_state():613] send defer: 12 2024-05-25 23:39:42,331 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: defer 2024-05-25 23:39:42,331 INFO HandlerThread:43320 [handler.py:handle_request_defer():184] handle defer: 12 2024-05-25 23:39:42,331 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: defer 2024-05-25 23:39:42,331 INFO SenderThread:43320 [sender.py:send_request_defer():609] handle sender defer: 12 2024-05-25 23:39:42,331 INFO SenderThread:43320 [file_stream.py:finish():601] file stream finish called 2024-05-25 23:39:42,607 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: poll_exit 2024-05-25 23:39:42,730 INFO SenderThread:43320 [file_stream.py:finish():605] file stream finish is done 2024-05-25 23:39:42,730 INFO SenderThread:43320 [sender.py:transition_state():613] send defer: 13 2024-05-25 23:39:42,730 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: poll_exit 2024-05-25 23:39:42,731 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: defer 2024-05-25 23:39:42,731 INFO HandlerThread:43320 [handler.py:handle_request_defer():184] handle defer: 13 2024-05-25 23:39:42,731 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: defer 2024-05-25 23:39:42,731 INFO SenderThread:43320 [sender.py:send_request_defer():609] handle sender defer: 13 2024-05-25 23:39:42,731 INFO SenderThread:43320 [sender.py:transition_state():613] send defer: 14 2024-05-25 23:39:42,731 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: defer 2024-05-25 23:39:42,731 INFO HandlerThread:43320 [handler.py:handle_request_defer():184] handle defer: 14 2024-05-25 23:39:42,731 DEBUG SenderThread:43320 [sender.py:send():378] send: final 2024-05-25 23:39:42,731 DEBUG SenderThread:43320 [sender.py:send():378] send: footer 2024-05-25 23:39:42,731 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: defer 2024-05-25 23:39:42,731 INFO SenderThread:43320 [sender.py:send_request_defer():609] handle sender defer: 14 2024-05-25 23:39:42,732 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: poll_exit 2024-05-25 23:39:42,732 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: poll_exit 2024-05-25 23:39:42,732 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: internal_messages 2024-05-25 23:39:42,733 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: server_info 2024-05-25 23:39:42,733 DEBUG SenderThread:43320 [sender.py:send_request():405] send_request: server_info 2024-05-25 23:39:42,774 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: get_summary 2024-05-25 23:39:42,774 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: sampled_history 2024-05-25 23:39:42,862 DEBUG HandlerThread:43320 [handler.py:handle_request():158] handle_request: shutdown 2024-05-25 23:39:42,862 INFO HandlerThread:43320 [handler.py:finish():882] shutting down handler 2024-05-25 23:39:43,733 INFO WriterThread:43320 [datastore.py:close():296] close: /root/mistral-finetune/runseed5/wandb/run-20240525_214643-cngfo6vp/run-cngfo6vp.wandb 2024-05-25 23:39:43,861 INFO SenderThread:43320 [sender.py:finish():1545] shutting down sender 2024-05-25 23:39:43,861 INFO SenderThread:43320 [file_pusher.py:finish():169] shutting down file pusher 2024-05-25 23:39:43,861 INFO SenderThread:43320 [file_pusher.py:join():175] waiting for file pusher