| 2024-04-26 15:57:15,698 INFO StreamThr :713738 [internal.py:wandb_internal():86] W&B internal server running at pid: 713738, started at: 2024-04-26 15:57:15.697145 | |
| 2024-04-26 15:57:15,700 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status | |
| 2024-04-26 15:57:15,705 INFO WriterThread:713738 [datastore.py:open_for_write():87] open: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/run-vljs8xzo.wandb | |
| 2024-04-26 15:57:15,706 DEBUG SenderThread:713738 [sender.py:send():379] send: header | |
| 2024-04-26 15:57:15,754 DEBUG SenderThread:713738 [sender.py:send():379] send: run | |
| 2024-04-26 15:57:16,079 INFO SenderThread:713738 [dir_watcher.py:__init__():211] watching files in: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files | |
| 2024-04-26 15:57:16,079 INFO SenderThread:713738 [sender.py:_start_run_threads():1124] run started: vljs8xzo with start time 1714147035.700869 | |
| 2024-04-26 15:57:16,085 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: check_version | |
| 2024-04-26 15:57:16,086 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: check_version | |
| 2024-04-26 15:57:16,171 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: run_start | |
| 2024-04-26 15:57:16,229 DEBUG HandlerThread:713738 [system_info.py:__init__():26] System info init | |
| 2024-04-26 15:57:16,229 DEBUG HandlerThread:713738 [system_info.py:__init__():41] System info init done | |
| 2024-04-26 15:57:16,229 INFO HandlerThread:713738 [system_monitor.py:start():194] Starting system monitor | |
| 2024-04-26 15:57:16,229 INFO SystemMonitor:713738 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2024-04-26 15:57:16,229 INFO HandlerThread:713738 [system_monitor.py:probe():214] Collecting system info | |
| 2024-04-26 15:57:16,230 INFO SystemMonitor:713738 [interfaces.py:start():190] Started cpu monitoring | |
| 2024-04-26 15:57:16,230 INFO SystemMonitor:713738 [interfaces.py:start():190] Started disk monitoring | |
| 2024-04-26 15:57:16,230 INFO SystemMonitor:713738 [interfaces.py:start():190] Started gpu monitoring | |
| 2024-04-26 15:57:16,231 INFO SystemMonitor:713738 [interfaces.py:start():190] Started memory monitoring | |
| 2024-04-26 15:57:16,231 INFO SystemMonitor:713738 [interfaces.py:start():190] Started network monitoring | |
| 2024-04-26 15:57:16,277 DEBUG HandlerThread:713738 [system_info.py:probe():150] Probing system | |
| 2024-04-26 15:57:16,279 DEBUG HandlerThread:713738 [system_info.py:_probe_git():135] Probing git | |
| 2024-04-26 15:57:16,299 DEBUG HandlerThread:713738 [system_info.py:_probe_git():143] Probing git done | |
| 2024-04-26 15:57:16,300 DEBUG HandlerThread:713738 [system_info.py:probe():198] Probing system done | |
| 2024-04-26 15:57:16,300 DEBUG HandlerThread:713738 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-1048-aws-x86_64-with-glibc2.31', 'python': '3.11.9', 'heartbeatAt': '2024-04-26T15:57:16.277357', 'startedAt': '2024-04-26T15:57:15.677140', 'docker': None, 'cuda': None, 'args': ('./config_full.yaml',), 'state': 'running', 'program': '/fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/./run_dpo.py', 'codePathLocal': 'run_dpo.py', 'codePath': 'run_dpo.py', 'git': {'remote': 'https://huggingface.co/sanchit-gandhi/distil-zephyr-1.5b-dpo-ultrafeedback', 'commit': '087b22a9416f56b8862e6f5259874a73b30a9c0b'}, 'email': None, 'root': '/fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback', 'host': 'ip-26-0-160-225', 'username': 'sanchit', 'executable': '/fsx/sanchit/miniconda3/envs/alignment/bin/python', 'cpu_count': 96, 'cpu_count_logical': 96, 'cpu_freq': {'current': 2728.6554687499997, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3569.387, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.689, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3598.118, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.248, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.735, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3598.023, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3594.821, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}, {'current': 3597.904, 'min': 0.0, 'max': 0.0}, {'current': 2650.0, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 290.7472343444824, 'used': 59.15196990966797}}, 'gpu': 'NVIDIA H100 80GB HBM3', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}, {'name': 'NVIDIA H100 80GB HBM3', 'memory_total': 85520809984}], 'memory': {'total': 1999.985538482666}} | |
| 2024-04-26 15:57:16,300 INFO HandlerThread:713738 [system_monitor.py:probe():224] Finished collecting system info | |
| 2024-04-26 15:57:16,300 INFO HandlerThread:713738 [system_monitor.py:probe():227] Publishing system info | |
| 2024-04-26 15:57:16,300 DEBUG HandlerThread:713738 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment | |
| 2024-04-26 15:57:17,081 INFO Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/conda-environment.yaml | |
| 2024-04-26 15:57:20,083 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/conda-environment.yaml | |
| 2024-04-26 15:57:20,172 DEBUG HandlerThread:713738 [system_info.py:_save_conda():222] Saving conda packages done | |
| 2024-04-26 15:57:20,174 INFO HandlerThread:713738 [system_monitor.py:probe():229] Finished publishing system info | |
| 2024-04-26 15:57:20,199 DEBUG SenderThread:713738 [sender.py:send():379] send: files | |
| 2024-04-26 15:57:20,199 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now | |
| 2024-04-26 15:57:20,348 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: python_packages | |
| 2024-04-26 15:57:20,349 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: python_packages | |
| 2024-04-26 15:57:20,349 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 15:57:20,351 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 15:57:20,352 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 15:57:20,453 DEBUG SenderThread:713738 [sender.py:send():379] send: telemetry | |
| 2024-04-26 15:57:20,453 DEBUG SenderThread:713738 [sender.py:send():379] send: config | |
| 2024-04-26 15:57:20,454 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:57:20,454 DEBUG SenderThread:713738 [sender.py:send():379] send: telemetry | |
| 2024-04-26 15:57:20,454 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:57:20,454 WARNING SenderThread:713738 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen) | |
| 2024-04-26 15:57:20,454 DEBUG SenderThread:713738 [sender.py:send():379] send: telemetry | |
| 2024-04-26 15:57:20,491 INFO wandb-upload_0:713738 [upload_job.py:push():131] Uploaded file /tmp/tmptr8aibf2wandb/74m4dddf-wandb-metadata.json | |
| 2024-04-26 15:57:21,085 INFO Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:21,085 INFO Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/requirements.txt | |
| 2024-04-26 15:57:21,086 INFO Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-metadata.json | |
| 2024-04-26 15:57:21,456 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:57:22,512 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-26 15:57:22,515 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:57:22,516 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:57:22,516 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:57:22,516 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:57:22,516 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:57:22,516 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:57:22,516 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:57:22,516 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:57:22,517 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:57:22,517 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:57:22,517 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:57:22,517 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:57:22,517 DEBUG SenderThread:713738 [sender.py:send():379] send: history | |
| 2024-04-26 15:57:22,517 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-26 15:57:22,519 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-26 15:57:23,088 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:23,089 INFO Thread-12 :713738 [dir_watcher.py:_on_file_created():271] file/dir created: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json | |
| 2024-04-26 15:57:25,091 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:27,096 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:27,303 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:57:29,100 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:31,102 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:32,406 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:57:33,103 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:35,106 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:35,349 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 15:57:35,350 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 15:57:35,351 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 15:57:37,110 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:37,525 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:57:39,114 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:41,117 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:42,595 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:57:43,121 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:45,126 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:47,130 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:47,734 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:57:48,135 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/config.yaml | |
| 2024-04-26 15:57:49,137 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:50,349 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 15:57:50,350 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 15:57:50,351 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 15:57:51,141 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:52,879 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:57:53,145 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:53,204 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-26 15:57:53,206 DEBUG SenderThread:713738 [sender.py:send():379] send: history | |
| 2024-04-26 15:57:53,206 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-26 15:57:53,208 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-26 15:57:54,151 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json | |
| 2024-04-26 15:57:55,153 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:57,155 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:57:58,052 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:57:59,157 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:01,161 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:03,163 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:58:03,166 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:05,169 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:05,349 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 15:58:05,349 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 15:58:05,352 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 15:58:07,172 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:08,289 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:58:09,174 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:11,176 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:13,178 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:13,439 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:58:15,180 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:16,231 DEBUG SystemMonitor:713738 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2024-04-26 15:58:16,235 DEBUG SenderThread:713738 [sender.py:send():379] send: stats | |
| 2024-04-26 15:58:17,182 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:18,586 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:58:19,184 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:20,349 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 15:58:20,349 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 15:58:20,352 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 15:58:21,187 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:23,190 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:23,691 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:58:25,192 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:25,356 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-26 15:58:25,358 DEBUG SenderThread:713738 [sender.py:send():379] send: history | |
| 2024-04-26 15:58:25,359 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-26 15:58:25,360 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-26 15:58:26,195 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json | |
| 2024-04-26 15:58:27,197 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:28,906 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:58:29,201 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:31,205 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:33,210 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:34,031 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:58:35,214 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:35,349 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 15:58:35,349 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 15:58:35,352 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 15:58:37,219 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:39,148 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:58:39,223 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:41,228 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:43,232 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:44,292 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:58:45,237 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:46,237 DEBUG SenderThread:713738 [sender.py:send():379] send: stats | |
| 2024-04-26 15:58:47,239 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:49,241 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:49,372 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:58:50,350 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 15:58:50,350 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 15:58:50,351 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 15:58:51,243 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:53,247 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:54,523 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:58:55,252 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:57,254 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:57,437 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-26 15:58:57,439 DEBUG SenderThread:713738 [sender.py:send():379] send: history | |
| 2024-04-26 15:58:57,440 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-26 15:58:57,441 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-26 15:58:58,256 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json | |
| 2024-04-26 15:58:59,257 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:58:59,663 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:59:01,259 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:03,261 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:04,824 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:59:05,263 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:05,349 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 15:59:05,350 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 15:59:05,351 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 15:59:07,265 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:09,267 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:09,968 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:59:11,269 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:13,271 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:15,260 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:59:15,273 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:16,238 DEBUG SenderThread:713738 [sender.py:send():379] send: stats | |
| 2024-04-26 15:59:17,285 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:19,288 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:20,350 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 15:59:20,350 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 15:59:20,352 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 15:59:20,496 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:59:21,290 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:23,292 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:25,294 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:25,498 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:59:27,296 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:29,297 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:29,411 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-26 15:59:29,413 DEBUG SenderThread:713738 [sender.py:send():379] send: history | |
| 2024-04-26 15:59:29,413 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-26 15:59:29,415 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-26 15:59:30,299 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json | |
| 2024-04-26 15:59:30,708 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:59:31,300 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:33,302 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:35,304 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:35,350 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 15:59:35,350 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 15:59:35,352 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 15:59:35,807 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:59:37,306 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:37,698 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-26 15:59:37,700 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:59:37,701 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:59:37,701 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:59:37,702 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:59:37,702 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:59:37,702 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:59:37,702 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:59:37,702 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:59:37,702 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:59:37,703 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:59:37,703 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:59:37,703 DEBUG SenderThread:713738 [sender.py:send():379] send: metric | |
| 2024-04-26 15:59:37,703 DEBUG SenderThread:713738 [sender.py:send():379] send: history | |
| 2024-04-26 15:59:37,703 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-26 15:59:37,705 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-26 15:59:38,308 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json | |
| 2024-04-26 15:59:39,309 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:41,714 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:59:46,241 DEBUG SenderThread:713738 [sender.py:send():379] send: stats | |
| 2024-04-26 15:59:47,242 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:59:49,320 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 15:59:51,474 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 15:59:51,474 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 15:59:51,734 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 15:59:52,625 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 15:59:53,325 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/config.yaml | |
| 2024-04-26 15:59:57,725 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:00:02,726 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:00:05,958 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 16:00:06,058 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 16:00:06,059 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 16:00:08,145 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:00:09,342 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:11,344 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:13,346 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:13,350 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:00:15,348 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:16,254 DEBUG SenderThread:713738 [sender.py:send():379] send: stats | |
| 2024-04-26 16:00:17,350 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:18,400 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:00:19,352 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:20,551 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 16:00:20,552 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 16:00:20,552 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 16:00:21,354 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:23,356 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:23,539 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:00:25,358 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:27,360 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:29,363 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:29,525 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:00:31,365 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:33,367 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:34,629 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:00:35,369 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:35,552 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 16:00:35,552 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 16:00:35,553 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 16:00:37,371 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:38,753 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-26 16:00:38,755 DEBUG SenderThread:713738 [sender.py:send():379] send: history | |
| 2024-04-26 16:00:38,755 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-26 16:00:38,757 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-26 16:00:39,374 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:39,374 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json | |
| 2024-04-26 16:00:39,757 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:00:41,376 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:43,378 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:44,796 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:00:45,380 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:46,244 DEBUG SenderThread:713738 [sender.py:send():379] send: stats | |
| 2024-04-26 16:00:47,382 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:49,384 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:49,828 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:00:50,551 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 16:00:50,552 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 16:00:50,552 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 16:00:51,386 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:53,388 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:54,857 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:00:55,390 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:57,392 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:00:59,393 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:00,017 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:01:01,396 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:03,398 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:05,177 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:01:05,400 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:05,552 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 16:01:05,552 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 16:01:05,552 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 16:01:07,402 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:09,404 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:10,447 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:01:10,508 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-26 16:01:10,510 DEBUG SenderThread:713738 [sender.py:send():379] send: history | |
| 2024-04-26 16:01:10,511 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-26 16:01:10,512 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-26 16:01:11,406 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:11,406 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json | |
| 2024-04-26 16:01:13,408 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:15,410 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:15,586 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:01:16,246 DEBUG SenderThread:713738 [sender.py:send():379] send: stats | |
| 2024-04-26 16:01:17,412 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:19,414 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:20,552 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 16:01:20,553 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 16:01:20,553 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 16:01:21,031 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:01:21,416 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:23,418 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:25,420 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:26,828 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:01:27,422 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:29,424 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:31,426 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:31,991 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:01:33,428 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:35,430 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:35,552 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 16:01:35,553 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 16:01:35,554 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 16:01:37,138 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:01:37,432 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:39,434 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:41,436 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:42,208 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:01:42,568 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-26 16:01:42,570 DEBUG SenderThread:713738 [sender.py:send():379] send: history | |
| 2024-04-26 16:01:42,570 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-26 16:01:42,573 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-26 16:01:43,439 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:43,439 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json | |
| 2024-04-26 16:01:45,441 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:46,260 DEBUG SenderThread:713738 [sender.py:send():379] send: stats | |
| 2024-04-26 16:01:47,409 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:01:47,444 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:49,446 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:50,552 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 16:01:50,553 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 16:01:50,553 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 16:01:51,448 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:52,597 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:01:53,450 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:55,451 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:57,454 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:01:57,688 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:01:59,455 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:02:01,457 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:02:02,736 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:02:03,459 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:02:05,461 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:02:05,552 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 16:02:05,553 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 16:02:05,553 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 16:02:07,463 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:02:08,031 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:02:09,465 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:02:11,467 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:02:13,147 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:02:13,469 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:02:14,469 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-26 16:02:14,472 DEBUG SenderThread:713738 [sender.py:send():379] send: history | |
| 2024-04-26 16:02:14,472 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-26 16:02:14,473 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-26 16:02:15,474 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:02:15,475 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json | |
| 2024-04-26 16:02:16,250 DEBUG SenderThread:713738 [sender.py:send():379] send: stats | |
| 2024-04-26 16:02:17,476 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:02:18,328 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:02:19,478 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:02:20,553 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 16:02:20,553 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 16:02:20,553 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 16:02:21,480 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:02:22,759 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history | |
| 2024-04-26 16:02:22,762 DEBUG SenderThread:713738 [sender.py:send():379] send: history | |
| 2024-04-26 16:02:22,763 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record | |
| 2024-04-26 16:02:22,764 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end | |
| 2024-04-26 16:02:23,483 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:02:23,483 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json | |
| 2024-04-26 16:02:23,774 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:02:25,485 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:02:28,775 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:02:34,173 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:02:35,496 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log | |
| 2024-04-26 16:02:36,046 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 16:02:36,196 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 16:02:36,197 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |
| 2024-04-26 16:02:39,290 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:02:44,290 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:02:46,253 DEBUG SenderThread:713738 [sender.py:send():379] send: stats | |
| 2024-04-26 16:02:50,254 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report | |
| 2024-04-26 16:02:50,703 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status | |
| 2024-04-26 16:02:50,703 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2024-04-26 16:02:50,704 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status | |