2024-04-26 15:57:15,698 INFO    StreamThr :713738 [internal.py:wandb_internal():86] W&B internal server running at pid: 713738, started at: 2024-04-26 15:57:15.697145
2024-04-26 15:57:15,700 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status
2024-04-26 15:57:15,705 INFO    WriterThread:713738 [datastore.py:open_for_write():87] open: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/run-vljs8xzo.wandb
2024-04-26 15:57:15,706 DEBUG   SenderThread:713738 [sender.py:send():379] send: header
2024-04-26 15:57:15,754 DEBUG   SenderThread:713738 [sender.py:send():379] send: run
2024-04-26 15:57:16,079 INFO    SenderThread:713738 [dir_watcher.py:__init__():211] watching files in: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files
2024-04-26 15:57:16,079 INFO    SenderThread:713738 [sender.py:_start_run_threads():1124] run started: vljs8xzo with start time 1714147035.700869
2024-04-26 15:57:16,085 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: check_version
2024-04-26 15:57:16,086 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: check_version
2024-04-26 15:57:16,171 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: run_start
2024-04-26 15:57:16,229 DEBUG   HandlerThread:713738 [system_info.py:__init__():26] System info init
2024-04-26 15:57:16,229 DEBUG   HandlerThread:713738 [system_info.py:__init__():41] System info init done
2024-04-26 15:57:16,229 INFO    HandlerThread:713738 [system_monitor.py:start():194] Starting system monitor
2024-04-26 15:57:16,229 INFO    SystemMonitor:713738 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-04-26 15:57:16,229 INFO    HandlerThread:713738 [system_monitor.py:probe():214] Collecting system info
2024-04-26 15:57:16,230 INFO    SystemMonitor:713738 [interfaces.py:start():190] Started cpu monitoring
2024-04-26 15:57:16,230 INFO    SystemMonitor:713738 [interfaces.py:start():190] Started disk monitoring
2024-04-26 15:57:16,230 INFO    SystemMonitor:713738 [interfaces.py:start():190] Started gpu monitoring
2024-04-26 15:57:16,231 INFO    SystemMonitor:713738 [interfaces.py:start():190] Started memory monitoring
2024-04-26 15:57:16,231 INFO    SystemMonitor:713738 [interfaces.py:start():190] Started network monitoring
2024-04-26 15:57:16,277 DEBUG   HandlerThread:713738 [system_info.py:probe():150] Probing system
2024-04-26 15:57:16,279 DEBUG   HandlerThread:713738 [system_info.py:_probe_git():135] Probing git
2024-04-26 15:57:16,299 DEBUG   HandlerThread:713738 [system_info.py:_probe_git():143] Probing git done
2024-04-26 15:57:16,300 DEBUG   HandlerThread:713738 [system_info.py:probe():198] Probing system done
2024-04-26 15:57:16,300 DEBUG   HandlerThread:713738 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-1048-aws-x86_64-with-glibc2.31', 'python': '3.11.9', 'heartbeatAt': '2024-04-26T15:57:16.277357', 'startedAt': '2024-04-26T15:57:15.677140', 'docker': None, 'cuda': None, 'args': ('./config_full.yaml',), 'state': 'running', 'program': '/fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/./run_dpo.py', 'codePathLocal': 'run_dpo.py', 'codePath': 'run_dpo.py', 'git': {'remote': 'https://huggingface.co/sanchit-gandhi/distil-zephyr-1.5b-dpo-ultrafeedback', 'commit': '087b22a9416f56b8862e6f5259874a73b30a9c0b'}, 'email': None, 'root': '/fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback', 'host': 'ip-26-0-160-225', 'username': 'sanchit', 'executable': '/fsx/sanchit/miniconda3/envs/alignment/bin/python', 'cpu_count': 96, 'cpu_count_logical': 96, 'cpu_freq': {'current': 2728.6554687499997, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3569.387, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.689, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3598.118, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.248, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.735, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3598.023, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3594.821, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.904, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 290.7472343444824, 'used': 59.15196990966797}}, 'gpu': 'NVIDIA H100 80GB HBM3', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}], 'memory': {'total': 1999.985538482666}}
2024-04-26 15:57:16,300 INFO    HandlerThread:713738 [system_monitor.py:probe():224] Finished collecting system info
2024-04-26 15:57:16,300 INFO    HandlerThread:713738 [system_monitor.py:probe():227] Publishing system info
2024-04-26 15:57:16,300 DEBUG   HandlerThread:713738 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-04-26 15:57:17,081 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/conda-environment.yaml
2024-04-26 15:57:20,083 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/conda-environment.yaml
2024-04-26 15:57:20,172 DEBUG   HandlerThread:713738 [system_info.py:_save_conda():222] Saving conda packages done
2024-04-26 15:57:20,174 INFO    HandlerThread:713738 [system_monitor.py:probe():229] Finished publishing system info
2024-04-26 15:57:20,199 DEBUG   SenderThread:713738 [sender.py:send():379] send: files
2024-04-26 15:57:20,199 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now
2024-04-26 15:57:20,348 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: python_packages
2024-04-26 15:57:20,349 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: python_packages
2024-04-26 15:57:20,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:57:20,351 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:57:20,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:57:20,453 DEBUG   SenderThread:713738 [sender.py:send():379] send: telemetry
2024-04-26 15:57:20,453 DEBUG   SenderThread:713738 [sender.py:send():379] send: config
2024-04-26 15:57:20,454 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:20,454 DEBUG   SenderThread:713738 [sender.py:send():379] send: telemetry
2024-04-26 15:57:20,454 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:20,454 WARNING SenderThread:713738 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen)
2024-04-26 15:57:20,454 DEBUG   SenderThread:713738 [sender.py:send():379] send: telemetry
2024-04-26 15:57:20,491 INFO    wandb-upload_0:713738 [upload_job.py:push():131] Uploaded file /tmp/tmptr8aibf2wandb/74m4dddf-wandb-metadata.json
2024-04-26 15:57:21,085 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:21,085 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/requirements.txt
2024-04-26 15:57:21,086 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-metadata.json
2024-04-26 15:57:21,456 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:22,512 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:57:22,515 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,516 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:57:22,517 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:57:22,519 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:57:23,088 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:23,089 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:57:25,091 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:27,096 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:27,303 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:29,100 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:31,102 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:32,406 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:33,103 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:35,106 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:35,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:57:35,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:57:35,351 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:57:37,110 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:37,525 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:39,114 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:41,117 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:42,595 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:43,121 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:45,126 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:47,130 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:47,734 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:48,135 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/config.yaml
2024-04-26 15:57:49,137 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:50,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:57:50,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:57:50,351 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:57:51,141 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:52,879 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:53,145 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:53,204 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:57:53,206 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:57:53,206 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:57:53,208 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:57:54,151 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:57:55,153 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:57,155 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:57:58,052 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:57:59,157 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:01,161 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:03,163 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:03,166 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:05,169 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:05,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:58:05,349 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:58:05,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:58:07,172 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:08,289 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:09,174 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:11,176 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:13,178 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:13,439 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:15,180 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:16,231 DEBUG   SystemMonitor:713738 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-04-26 15:58:16,235 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 15:58:17,182 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:18,586 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:19,184 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:20,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:58:20,349 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:58:20,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:58:21,187 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:23,190 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:23,691 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:25,192 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:25,356 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:58:25,358 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:58:25,359 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:58:25,360 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:58:26,195 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:58:27,197 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:28,906 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:29,201 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:31,205 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:33,210 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:34,031 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:35,214 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:35,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:58:35,349 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:58:35,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:58:37,219 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:39,148 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:39,223 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:41,228 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:43,232 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:44,292 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:45,237 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:46,237 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 15:58:47,239 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:49,241 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:49,372 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:50,350 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:58:50,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:58:50,351 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:58:51,243 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:53,247 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:54,523 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:58:55,252 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:57,254 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:57,437 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:58:57,439 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:58:57,440 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:58:57,441 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:58:58,256 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:58:59,257 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:58:59,663 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:01,259 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:03,261 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:04,824 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:05,263 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:05,349 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:59:05,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:59:05,351 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:59:07,265 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:09,267 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:09,968 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:11,269 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:13,271 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:15,260 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:15,273 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:16,238 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 15:59:17,285 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:19,288 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:20,350 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:59:20,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:59:20,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:59:20,496 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:21,290 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:23,292 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:25,294 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:25,498 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:27,296 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:29,297 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:29,411 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:59:29,413 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:59:29,413 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:59:29,415 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:59:30,299 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:59:30,708 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:31,300 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:33,302 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:35,304 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:35,350 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:59:35,350 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:59:35,352 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:59:35,807 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:37,306 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:37,698 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
2024-04-26 15:59:37,700 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,701 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,701 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,702 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,703 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,703 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,703 DEBUG   SenderThread:713738 [sender.py:send():379] send: metric
2024-04-26 15:59:37,703 DEBUG   SenderThread:713738 [sender.py:send():379] send: history
2024-04-26 15:59:37,703 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
2024-04-26 15:59:37,705 INFO    SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-04-26 15:59:38,308 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
2024-04-26 15:59:39,309 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:41,714 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:46,241 DEBUG   SenderThread:713738 [sender.py:send():379] send: stats
2024-04-26 15:59:47,242 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:49,320 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
2024-04-26 15:59:51,474 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 15:59:51,474 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
2024-04-26 15:59:51,734 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 15:59:52,625 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 15:59:53,325 INFO    Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/config.yaml
2024-04-26 15:59:57,725 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:00:02,726 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
2024-04-26 16:00:05,958 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
2024-04-26 16:00:06,058 DEBUG   HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
2024-04-26 16:00:06,059 DEBUG   SenderThread:713738 [sender.py:send_request():406] send_request: stop_status