2024-04-26 15:57:15,698 INFO    StreamThr :713738 [internal.py:wandb_internal():86] W&B internal server running at pid: 713738, started at: 2024-04-26 15:57:15.697145
2024-04-26 15:57:15,700 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status
2024-04-26 15:57:15,705 INFO    WriterThread:713738 [datastore.py:open_for_write():87] open: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/run-vljs8xzo.wandb
2024-04-26 15:57:15,706 DEBUG   SenderThread:713738 [sender.py:send():379] send: header
2024-04-26 15:57:15,754 DEBUG   SenderThread:713738 [sender.py:send():379] send: run
2024-04-26 15:57:16,079 INFO    SenderThread:713738 [dir_watcher.py:__init__():211] watching files in: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files
2024-04-26 15:57:16,079 INFO    SenderThread:713738 [sender.py:_start_run_threads():1124] run started: vljs8xzo with start time 1714147035.700869
2024-04-26 15:57:16,085 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: check_version
2024-04-26 15:57:16,086 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: check_version
2024-04-26 15:57:16,171 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: run_start
2024-04-26 15:57:16,229 DEBUG   HandlerThread:713738 [system_info.py:__init__():26] System info init
2024-04-26 15:57:16,229 DEBUG   HandlerThread:713738 [system_info.py:__init__():41] System info init done
2024-04-26 15:57:16,229 INFO    HandlerThread:713738 [system_monitor.py:start():194] Starting system monitor
2024-04-26 15:57:16,229 INFO    SystemMonitor:713738 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-04-26 15:57:16,229 INFO    HandlerThread:713738 [system_monitor.py:probe():214] Collecting system info
2024-04-26 15:57:16,230 INFO    SystemMonitor:713738 [interfaces.py:start():190] Started cpu monitoring
2024-04-26 15:57:16,230 INFO    SystemMonitor:713738 [interfaces.py:start():190] Started disk monitoring
2024-04-26 15:57:16,230 INFO    SystemMonitor:713738 [interfaces.py:start():190] Started gpu monitoring
2024-04-26 15:57:16,231 INFO    SystemMonitor:713738 [interfaces.py:start():190] Started memory monitoring
2024-04-26 15:57:16,231 INFO    SystemMonitor:713738 [interfaces.py:start():190] Started network monitoring
2024-04-26 15:57:16,277 DEBUG   HandlerThread:713738 [system_info.py:probe():150] Probing system
2024-04-26 15:57:16,279 DEBUG   HandlerThread:713738 [system_info.py:_probe_git():135] Probing git
2024-04-26 15:57:16,299 DEBUG   HandlerThread:713738 [system_info.py:_probe_git():143] Probing git done
2024-04-26 15:57:16,300 DEBUG   HandlerThread:713738 [system_info.py:probe():198] Probing system done
2024-04-26 15:57:16,300 DEBUG   HandlerThread:713738 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-1048-aws-x86_64-with-glibc2.31', 'python': '3.11.9', 'heartbeatAt': '2024-04-26T15:57:16.277357', 'startedAt': '2024-04-26T15:57:15.677140', 'docker': None, 'cuda': None, 'args': ('./config_full.yaml',), 'state': 'running', 'program': '/fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/./run_dpo.py', 'codePathLocal': 'run_dpo.py', 'codePath': 'run_dpo.py', 'git': {'remote': 'https://huggingface.co/sanchit-gandhi/distil-zephyr-1.5b-dpo-ultrafeedback', 'commit': '087b22a9416f56b8862e6f5259874a73b30a9c0b'}, 'email': None, 'root': '/fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback', 'host': 'ip-26-0-160-225', 'username': 'sanchit', 'executable': '/fsx/sanchit/miniconda3/envs/alignment/bin/python', 'cpu_count': 96, 'cpu_count_logical': 96, 'cpu_freq': {'current': 2728.6554687499997, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3569.387, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.689, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3598.118, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.248, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.735, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3598.023, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3594.821, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.904, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 290.7472343444824, 'used': 59.15196990966797}}, 'gpu': 'NVIDIA H100 80GB HBM3', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}], 'memory': {'total': 1999.985538482666}}
2024-04-26 15:57:16,300 INFO    HandlerThread:713738 [system_monitor.py:probe():224] Finished collecting system info
2024-04-26 15:57:16,300 INFO    HandlerThread:713738 [system_monitor.py:probe():227] Publishing system info
2024-04-26 15:57:16,300 DEBUG   HandlerThread:713738 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-04-26 15:57:17,081 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/conda-environment.yaml
2024-04-26 15:57:20,083 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/conda-environment.yaml
2024-04-26 15:57:20,172 DEBUG   HandlerThread:713738 [system_info.py:_save_conda():222] Saving conda packages done
2024-04-26 15:57:20,174 INFO    HandlerThread:713738 [system_monitor.py:probe():229] Finished publishing system info
2024-04-26 15:57:20,199 DEBUG   SenderThread:713738 [sender.py:send():379] send: files
2024-04-26 15:57:20,199 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now
2024-04-26 15:57:20,348 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: python_packages
2024-04-26 15:57:20,349 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: python_packages
2024-04-26 15:57:20,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:57:20,351 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:57:20,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:57:20,453 DEBUG   SenderThread:713738 [sender.py:send():379] send: telemetry
2024-04-26 15:57:20,453 DEBUG   SenderThread:713738 [sender.py:send():379] send: config
2024-04-26 15:57:20,454 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:20,454 DEBUG   SenderThread:713738 [sender.py:send():379] send: telemetry
2024-04-26 15:57:20,454 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:20,454 WARNING SenderThread:713738 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen)
2024-04-26 15:57:20,454 DEBUG   SenderThread:713738 [sender.py:send():379] send: telemetry
2024-04-26 15:57:20,491 INFO    wandb-upload_0:713738 [upload_job.py:push():131] Uploaded file /tmp/tmptr8aibf2wandb/74m4dddf-wandb-metadata.json
2024-04-26 15:57:21,085 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:21,085 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/requirements.txt
2024-04-26 15:57:21,086 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-metadata.json
2024-04-26 15:57:21,456 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:22,512 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:57:22,515 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:57:22,519 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:57:23,088 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:23,089 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:57:25,091 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:27,096 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:27,303 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:29,100 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:31,102 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:32,406 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:33,103 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:35,106 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:35,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:57:35,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:57:35,351 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:57:37,110 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:37,525 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:39,114 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:41,117 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:42,595 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:43,121 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:45,126 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:47,130 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:47,734 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:48,135 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/config.yaml
2024-04-26 15:57:49,137 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:50,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:57:50,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:57:50,351 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:57:51,141 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:52,879 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:53,145 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:53,204 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:57:53,206 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:57:53,206 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:57:53,208 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:57:54,151 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:57:55,153 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:57,155 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:58,052 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:59,157 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:01,161 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:03,163 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:03,166 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:05,169 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:05,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:58:05,349 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:58:05,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:58:07,172 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:08,289 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:09,174 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:11,176 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:13,178 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:13,439 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:15,180 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:16,231 DEBUG   SystemMonitor:713738 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-04-26 15:58:16,235 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 15:58:17,182 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:18,586 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:19,184 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:20,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:58:20,349 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:58:20,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:58:21,187 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:23,190 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:23,691 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:25,192 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:25,356 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:58:25,358 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:58:25,359 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:58:25,360 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:58:26,195 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:58:27,197 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:28,906 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:29,201 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:31,205 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:33,210 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:34,031 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:35,214 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:35,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:58:35,349 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:58:35,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:58:37,219 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:39,148 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:39,223 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:41,228 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:43,232 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:44,292 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:45,237 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:46,237 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 15:58:47,239 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:49,241 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:49,372 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:50,350 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:58:50,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:58:50,351 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:58:51,243 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:53,247 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:54,523 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:55,252 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:57,254 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:57,437 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:58:57,439 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:58:57,440 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:58:57,441 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:58:58,256 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:58:59,257 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:59,663 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:01,259 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:03,261 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:04,824 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:05,263 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:05,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:59:05,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:59:05,351 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:59:07,265 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:09,267 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:09,968 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:11,269 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:13,271 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:15,260 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:15,273 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:16,238 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 15:59:17,285 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:19,288 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:20,350 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:59:20,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:59:20,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:59:20,496 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:21,290 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:23,292 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:25,294 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:25,498 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:27,296 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:29,297 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:29,411 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:59:29,413 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:59:29,413 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:59:29,415 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:59:30,299 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:59:30,708 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:31,300 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:33,302 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:35,304 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:35,350 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:59:35,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:59:35,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:59:35,807 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:37,306 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:37,698 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:59:37,700 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,701 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,701 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,703 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,703 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,703 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,703 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:59:37,703 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:59:37,705 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:59:38,308 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:59:39,309 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:41,714 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:46,241 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 15:59:47,242 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:49,320 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:51,474 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:59:51,474 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:59:51,734 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:59:52,625 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:53,325 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/config.yaml
2024-04-26 15:59:57,725 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:00:02,726 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:00:05,958 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:00:06,058 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:00:06,059 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:00:08,145 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:00:09,342 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:11,344 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:13,346 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:13,350 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:00:15,348 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:16,254 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:00:17,350 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:18,400 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:00:19,352 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:20,551 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:00:20,552 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:00:20,552 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:00:21,354 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:23,356 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:23,539 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:00:25,358 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:27,360 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:29,363 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:29,525 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:00:31,365 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:33,367 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:34,629 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:00:35,369 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:35,552 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:00:35,552 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:00:35,553 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:00:37,371 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:38,753 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:00:38,755 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:00:38,755 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:00:38,757 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:00:39,374 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:39,374 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:00:39,757 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:00:41,376 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:43,378 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:44,796 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:00:45,380 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:46,244 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:00:47,382 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:49,384 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:49,828 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:00:50,551 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:00:50,552 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:00:50,552 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:00:51,386 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:53,388 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:54,857 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:00:55,390 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:57,392 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:00:59,393 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:00,017 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:01:01,396 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:03,398 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:05,177 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:01:05,400 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:05,552 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:01:05,552 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:01:05,552 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:01:07,402 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:09,404 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:10,447 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:01:10,508 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:01:10,510 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:01:10,511 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:01:10,512 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:01:11,406 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:11,406 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:01:13,408 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:15,410 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:15,586 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:01:16,246 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:01:17,412 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:19,414 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:20,552 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:01:20,553 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:01:20,553 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:01:21,031 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:01:21,416 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:23,418 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:25,420 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:26,828 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:01:27,422 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:29,424 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:31,426 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:31,991 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:01:33,428 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:35,430 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:35,552 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:01:35,553 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:01:35,554 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:01:37,138 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:01:37,432 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:39,434 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:41,436 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:42,208 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:01:42,568 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:01:42,570 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:01:42,570 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:01:42,573 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:01:43,439 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:43,439 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:01:45,441 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:46,260 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:01:47,409 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:01:47,444 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:49,446 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:50,552 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:01:50,553 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:01:50,553 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:01:51,448 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:52,597 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:01:53,450 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:55,451 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:57,454 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:01:57,688 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:01:59,455 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:01,457 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:02,736 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:02:03,459 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:05,461 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:05,552 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:02:05,553 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:02:05,553 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:02:07,463 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:08,031 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:02:09,465 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:11,467 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:13,147 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:02:13,469 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:14,469 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:02:14,472 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:02:14,472 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:02:14,473 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:02:15,474 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:15,475 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:02:16,250 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:02:17,476 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:18,328 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:02:19,478 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:20,553 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:02:20,553 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:02:20,553 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:02:21,480 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:22,759 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:02:22,762 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:02:22,763 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:02:22,764 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:02:23,483 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:23,483 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:02:23,774 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:02:25,485 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:28,775 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:02:34,173 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:02:35,496 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:36,046 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:02:36,196 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:02:36,197 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:02:39,290 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:02:44,290 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:02:46,253 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:02:50,254 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:02:50,703 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:02:50,703 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:02:50,704 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:02:53,515 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:55,517 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:55,925 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:02:57,519 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:02:59,521 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:01,073 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:03:01,523 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:03,525 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:05,527 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:05,703 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:03:05,703 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:03:05,704 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:03:06,525 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:03:07,529 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:09,531 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:11,533 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:11,683 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:03:13,535 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:15,537 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:16,268 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:03:16,806 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:03:17,539 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:19,540 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:20,703 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:03:20,703 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:03:20,704 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:03:21,542 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:21,851 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:03:23,544 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:24,550 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:03:24,552 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:03:24,552 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:03:24,554 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:03:25,547 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:25,547 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:03:27,036 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:03:27,561 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:29,564 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:31,566 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:32,153 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:03:33,568 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:35,569 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:35,703 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:03:35,703 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:03:35,703 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:03:37,320 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:03:37,572 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:39,574 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:41,576 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:42,553 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:03:43,578 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:45,580 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:46,257 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:03:47,582 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:47,771 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:03:49,584 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:50,703 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:03:50,703 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:03:50,704 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:03:51,586 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:52,587 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:52,945 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:03:54,590 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:56,592 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:03:57,001 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:03:57,002 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:03:57,003 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:03:57,004 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:03:57,594 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:03:58,005 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:03:58,595 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:00,597 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:02,599 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:03,227 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:04:04,601 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:05,703 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:04:05,704 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:04:05,704 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:04:06,603 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:08,411 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:04:08,605 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:10,607 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:12,609 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:13,658 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:04:14,611 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:16,259 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:04:16,613 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:18,615 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:18,859 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:04:20,617 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:20,703 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:04:20,704 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:04:20,705 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:04:22,619 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:24,101 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:04:24,621 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:26,623 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:28,625 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:29,310 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:04:29,685 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:04:29,686 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:04:29,687 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:04:29,688 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:04:30,628 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:30,628 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:04:32,629 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:34,539 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:04:34,631 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:35,703 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:04:35,704 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:04:35,705 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:04:36,633 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:38,635 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:39,878 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:04:40,637 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:42,639 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:44,641 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:45,113 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:04:46,274 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:04:46,643 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:48,645 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:50,440 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:04:50,647 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:50,703 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:04:50,704 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:04:50,705 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:04:52,649 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:54,651 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:55,860 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:04:56,653 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:04:58,655 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:00,657 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:01,087 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:05:02,659 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:02,768 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:05:02,770 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:05:02,770 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:05:02,772 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:05:03,661 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:05:04,662 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:05,704 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:05:05,704 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:05:05,705 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:05:06,208 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:05:06,665 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:08,667 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:10,669 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:11,138 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:05:11,139 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:05:11,140 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:05:11,142 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:05:11,671 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:05:12,150 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:05:12,672 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:16,263 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:05:17,264 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:05:21,425 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:05:21,787 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:05:21,787 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:05:22,682 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:22,876 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:05:27,877 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:05:32,878 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:05:37,183 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:05:37,184 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:05:37,526 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:05:38,277 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:05:40,699 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:42,701 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:43,378 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:05:44,703 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:46,265 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:05:46,705 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:48,641 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:05:48,707 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:50,709 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:51,510 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:05:51,518 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:05:51,518 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:05:52,711 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:53,838 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:05:54,713 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:56,715 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:58,717 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:05:59,141 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:06:00,719 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:02,721 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:04,336 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:06:04,722 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:06,510 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:06:06,518 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:06:06,518 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:06:06,725 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:08,727 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:09,626 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:06:10,729 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:12,730 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:13,597 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:06:13,599 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:06:13,599 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:06:13,601 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:06:13,732 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:06:14,733 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:14,912 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:06:16,267 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:06:16,735 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:18,737 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:20,181 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:06:20,739 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:21,510 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:06:21,518 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:06:21,518 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:06:22,741 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:24,743 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:25,488 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:06:26,745 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:28,747 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:30,678 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:06:30,749 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:32,751 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:34,753 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:35,814 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:06:36,520 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:06:36,520 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:06:36,521 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:06:36,755 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:38,757 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:40,759 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:41,043 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:06:42,760 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:44,762 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:46,257 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:06:46,268 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:06:46,317 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:06:46,318 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:06:46,319 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:06:46,320 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:06:46,765 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:46,765 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:06:48,767 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:50,769 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:51,468 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:06:51,555 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:06:51,555 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:06:51,557 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:06:52,771 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:54,866 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:56,733 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:06:56,868 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:06:58,895 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:00,897 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:02,561 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:07:02,899 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:04,900 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:06,554 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:07:06,555 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:07:06,555 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:07:06,904 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:07,900 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:07:08,907 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:10,910 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:12,913 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:13,514 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:07:14,916 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:16,271 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:07:16,920 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:18,908 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:07:18,924 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:20,750 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:07:20,752 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:07:20,753 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:07:20,757 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:07:20,928 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:20,928 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:07:21,555 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:07:21,555 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:07:21,556 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:07:22,930 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:24,343 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:07:24,932 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:26,933 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:28,935 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:30,082 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:07:30,937 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:32,939 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:34,941 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:35,448 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:07:36,555 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:07:36,555 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:07:36,556 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:07:36,944 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:38,945 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:40,804 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:07:40,948 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:42,949 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:44,951 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:46,213 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:07:46,286 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:07:46,953 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:48,956 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:50,957 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:51,555 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:07:51,555 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:07:51,555 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:07:51,556 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:07:52,960 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:54,736 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:07:54,737 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:07:54,738 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:07:54,740 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:07:54,962 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:54,963 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:07:56,576 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:07:56,964 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:07:58,967 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:00,968 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:01,802 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:08:02,971 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:03,478 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:08:03,480 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:08:03,481 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:08:03,483 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:08:03,972 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:08:04,973 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:06,598 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:08:06,599 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:08:06,599 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:08:07,726 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:08:12,727 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:08:14,985 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:16,275 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:08:18,276 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:08:22,386 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:08:23,064 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:08:23,064 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:08:24,187 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:08:29,188 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:08:33,026 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:34,871 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:08:37,030 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:37,058 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:08:37,058 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:08:37,058 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:08:39,032 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:39,996 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:08:41,034 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:43,038 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:45,040 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:45,420 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:08:46,337 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:08:47,042 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:49,044 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:50,801 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:08:51,045 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:52,058 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:08:52,058 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:08:52,059 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:08:53,051 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:55,053 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:56,161 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:08:57,055 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:08:59,057 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:01,059 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:01,568 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:09:03,060 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:05,062 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:06,912 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:09:07,057 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:09:07,058 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:09:07,058 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:09:07,064 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:07,332 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:09:07,334 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:09:07,334 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:09:07,335 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:09:08,066 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:09:09,067 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:11,069 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:12,304 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:09:13,071 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:15,073 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:16,339 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:09:17,076 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:17,760 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:09:19,077 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:21,079 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:22,058 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:09:22,058 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:09:22,059 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:09:23,081 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:23,203 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:09:25,083 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:27,085 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:28,623 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:09:29,087 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:31,089 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:33,091 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:34,019 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:09:35,093 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:37,058 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:09:37,059 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:09:37,059 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:09:37,095 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:39,097 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:39,390 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:09:41,099 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:41,173 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:09:41,175 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:09:41,176 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:09:41,177 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:09:42,101 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:09:43,102 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:44,810 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:09:45,104 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:46,340 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:09:47,106 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:49,109 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:50,192 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:09:51,111 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:52,058 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:09:52,058 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:09:52,059 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:09:53,112 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:55,115 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:55,629 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:09:57,117 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:09:59,119 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:01,005 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:10:01,121 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:03,123 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:05,125 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:06,489 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:10:07,058 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:10:07,058 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:10:07,059 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:10:07,127 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:09,129 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:11,132 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:11,931 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:10:13,134 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:15,038 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:10:15,039 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:10:15,040 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:10:15,041 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:10:15,136 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:15,137 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:10:16,342 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:10:17,138 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:17,356 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:10:19,091 DEBUG   SenderThread:713738 [sender.py:send():379] send: telemetry
2024-04-26 16:10:19,093 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:10:19,094 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:10:19,097 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:10:19,098 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:10:19,099 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:10:19,099 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:10:19,101 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:10:19,101 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:10:19,102 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:10:19,103 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:10:19,104 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:10:19,105 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:10:19,105 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:10:19,106 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:10:19,141 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:19,141 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:10:21,143 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:22,059 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:10:22,059 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:10:22,059 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:10:22,467 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:10:23,145 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:25,147 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:27,149 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:27,423 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 16:10:27,425 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 16:10:27,425 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 16:10:27,429 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 16:10:28,151 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 16:10:28,437 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:10:29,153 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:29,154 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/config.yaml
2024-04-26 16:10:33,569 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:10:37,121 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:10:37,121 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:10:37,122 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:10:39,164 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:39,263 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:10:44,264 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:10:46,345 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:10:49,174 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:50,203 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:10:52,120 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:10:52,121 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:10:52,121 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:10:55,181 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:55,370 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:10:57,183 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:10:59,185 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:11:00,468 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:11:01,187 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:11:03,189 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:11:05,191 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:11:05,523 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:11:07,120 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:11:07,120 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:11:07,120 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:11:07,194 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:11:09,196 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:11:10,641 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:11:13,200 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:11:15,202 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:11:15,699 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:11:16,347 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:11:17,204 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:11:19,207 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:11:21,307 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:11:22,120 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:11:22,121 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:11:22,121 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:11:27,229 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:11:32,323 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:11:37,121 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:11:37,121 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:11:37,121 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:11:37,584 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:11:42,686 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:11:46,350 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:11:47,930 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:11:52,120 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:11:52,121 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:11:52,121 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:11:53,120 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:11:58,201 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:12:03,288 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:12:07,121 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:12:07,121 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:12:07,122 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:12:08,297 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:12:13,359 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:12:16,352 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 16:12:18,516 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:12:21,271 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:12:22,121 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:12:22,121 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:12:22,121 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 16:12:23,708 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:12:25,275 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:12:28,710 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:12:33,787 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:12:35,286 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 16:12:37,121 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:12:37,121 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:12:37,121 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status