Hugging Face's logo

sanchit-gandhi
/
distil-zephyr-1.5b-dpo-ultrafeedback

Text Generation
Transformers
TensorBoard
Safetensors
mistral
alignment-handbook
trl
dpo
Generated from Trainer
conversational
text-generation-inference
Inference Endpoints
Model card Files Metrics Community
sanchit-gandhi HF staff commited on
Commit
b703d56
•
1 Parent(s): 315a9a7

Training in progress, step 400

Browse files
Files changed (6) hide show
  1. runs/Apr26_15-48-43_ip-26-0-160-225/events.out.tfevents.1714147034.ip-26-0-160-225.711598.0 +2 -2
  2. wandb/debug-internal.log +170 -0
  3. wandb/run-20240426_155715-vljs8xzo/files/output.log +85 -0
  4. wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json +1 -1
  5. wandb/run-20240426_155715-vljs8xzo/logs/debug-internal.log +170 -0
  6. wandb/run-20240426_155715-vljs8xzo/run-vljs8xzo.wandb +0 -0
runs/Apr26_15-48-43_ip-26-0-160-225/events.out.tfevents.1714147034.ip-26-0-160-225.711598.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:001570658f281534496db82c62e1a044d09333fd849c089b8724f2efc43ffb43
3
- size 15854
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3a1fc27bf5185d63cd4e2dfcd363e9724966b1098397dd6e5f8a59a1a161d08c
3
+ size 19346
wandb/debug-internal.log CHANGED
@@ -577,3 +577,173 @@
577
  2024-04-26 16:05:37,184 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
578
  2024-04-26 16:05:37,526 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
579
  2024-04-26 16:05:38,277 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
577
  2024-04-26 16:05:37,184 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
578
  2024-04-26 16:05:37,526 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
579
  2024-04-26 16:05:38,277 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
580
+ 2024-04-26 16:05:40,699 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
581
+ 2024-04-26 16:05:42,701 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
582
+ 2024-04-26 16:05:43,378 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
583
+ 2024-04-26 16:05:44,703 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
584
+ 2024-04-26 16:05:46,265 DEBUG SenderThread:713738 [sender.py:send():379] send: stats
585
+ 2024-04-26 16:05:46,705 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
586
+ 2024-04-26 16:05:48,641 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
587
+ 2024-04-26 16:05:48,707 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
588
+ 2024-04-26 16:05:50,709 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
589
+ 2024-04-26 16:05:51,510 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
590
+ 2024-04-26 16:05:51,518 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
591
+ 2024-04-26 16:05:51,518 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
592
+ 2024-04-26 16:05:52,711 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
593
+ 2024-04-26 16:05:53,838 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
594
+ 2024-04-26 16:05:54,713 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
595
+ 2024-04-26 16:05:56,715 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
596
+ 2024-04-26 16:05:58,717 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
597
+ 2024-04-26 16:05:59,141 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
598
+ 2024-04-26 16:06:00,719 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
599
+ 2024-04-26 16:06:02,721 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
600
+ 2024-04-26 16:06:04,336 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
601
+ 2024-04-26 16:06:04,722 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
602
+ 2024-04-26 16:06:06,510 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
603
+ 2024-04-26 16:06:06,518 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
604
+ 2024-04-26 16:06:06,518 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
605
+ 2024-04-26 16:06:06,725 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
606
+ 2024-04-26 16:06:08,727 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
607
+ 2024-04-26 16:06:09,626 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
608
+ 2024-04-26 16:06:10,729 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
609
+ 2024-04-26 16:06:12,730 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
610
+ 2024-04-26 16:06:13,597 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
611
+ 2024-04-26 16:06:13,599 DEBUG SenderThread:713738 [sender.py:send():379] send: history
612
+ 2024-04-26 16:06:13,599 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
613
+ 2024-04-26 16:06:13,601 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
614
+ 2024-04-26 16:06:13,732 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
615
+ 2024-04-26 16:06:14,733 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
616
+ 2024-04-26 16:06:14,912 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
617
+ 2024-04-26 16:06:16,267 DEBUG SenderThread:713738 [sender.py:send():379] send: stats
618
+ 2024-04-26 16:06:16,735 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
619
+ 2024-04-26 16:06:18,737 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
620
+ 2024-04-26 16:06:20,181 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
621
+ 2024-04-26 16:06:20,739 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
622
+ 2024-04-26 16:06:21,510 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
623
+ 2024-04-26 16:06:21,518 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
624
+ 2024-04-26 16:06:21,518 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
625
+ 2024-04-26 16:06:22,741 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
626
+ 2024-04-26 16:06:24,743 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
627
+ 2024-04-26 16:06:25,488 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
628
+ 2024-04-26 16:06:26,745 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
629
+ 2024-04-26 16:06:28,747 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
630
+ 2024-04-26 16:06:30,678 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
631
+ 2024-04-26 16:06:30,749 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
632
+ 2024-04-26 16:06:32,751 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
633
+ 2024-04-26 16:06:34,753 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
634
+ 2024-04-26 16:06:35,814 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
635
+ 2024-04-26 16:06:36,520 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
636
+ 2024-04-26 16:06:36,520 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
637
+ 2024-04-26 16:06:36,521 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
638
+ 2024-04-26 16:06:36,755 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
639
+ 2024-04-26 16:06:38,757 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
640
+ 2024-04-26 16:06:40,759 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
641
+ 2024-04-26 16:06:41,043 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
642
+ 2024-04-26 16:06:42,760 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
643
+ 2024-04-26 16:06:44,762 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
644
+ 2024-04-26 16:06:46,257 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
645
+ 2024-04-26 16:06:46,268 DEBUG SenderThread:713738 [sender.py:send():379] send: stats
646
+ 2024-04-26 16:06:46,317 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
647
+ 2024-04-26 16:06:46,318 DEBUG SenderThread:713738 [sender.py:send():379] send: history
648
+ 2024-04-26 16:06:46,319 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
649
+ 2024-04-26 16:06:46,320 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
650
+ 2024-04-26 16:06:46,765 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
651
+ 2024-04-26 16:06:46,765 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
652
+ 2024-04-26 16:06:48,767 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
653
+ 2024-04-26 16:06:50,769 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
654
+ 2024-04-26 16:06:51,468 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
655
+ 2024-04-26 16:06:51,555 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
656
+ 2024-04-26 16:06:51,555 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
657
+ 2024-04-26 16:06:51,557 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
658
+ 2024-04-26 16:06:52,771 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
659
+ 2024-04-26 16:06:54,866 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
660
+ 2024-04-26 16:06:56,733 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
661
+ 2024-04-26 16:06:56,868 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
662
+ 2024-04-26 16:06:58,895 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
663
+ 2024-04-26 16:07:00,897 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
664
+ 2024-04-26 16:07:02,561 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
665
+ 2024-04-26 16:07:02,899 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
666
+ 2024-04-26 16:07:04,900 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
667
+ 2024-04-26 16:07:06,554 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
668
+ 2024-04-26 16:07:06,555 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
669
+ 2024-04-26 16:07:06,555 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
670
+ 2024-04-26 16:07:06,904 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
671
+ 2024-04-26 16:07:07,900 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
672
+ 2024-04-26 16:07:08,907 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
673
+ 2024-04-26 16:07:10,910 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
674
+ 2024-04-26 16:07:12,913 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
675
+ 2024-04-26 16:07:13,514 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
676
+ 2024-04-26 16:07:14,916 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
677
+ 2024-04-26 16:07:16,271 DEBUG SenderThread:713738 [sender.py:send():379] send: stats
678
+ 2024-04-26 16:07:16,920 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
679
+ 2024-04-26 16:07:18,908 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
680
+ 2024-04-26 16:07:18,924 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
681
+ 2024-04-26 16:07:20,750 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
682
+ 2024-04-26 16:07:20,752 DEBUG SenderThread:713738 [sender.py:send():379] send: history
683
+ 2024-04-26 16:07:20,753 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
684
+ 2024-04-26 16:07:20,757 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
685
+ 2024-04-26 16:07:20,928 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
686
+ 2024-04-26 16:07:20,928 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
687
+ 2024-04-26 16:07:21,555 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
688
+ 2024-04-26 16:07:21,555 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
689
+ 2024-04-26 16:07:21,556 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
690
+ 2024-04-26 16:07:22,930 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
691
+ 2024-04-26 16:07:24,343 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
692
+ 2024-04-26 16:07:24,932 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
693
+ 2024-04-26 16:07:26,933 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
694
+ 2024-04-26 16:07:28,935 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
695
+ 2024-04-26 16:07:30,082 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
696
+ 2024-04-26 16:07:30,937 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
697
+ 2024-04-26 16:07:32,939 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
698
+ 2024-04-26 16:07:34,941 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
699
+ 2024-04-26 16:07:35,448 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
700
+ 2024-04-26 16:07:36,555 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
701
+ 2024-04-26 16:07:36,555 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
702
+ 2024-04-26 16:07:36,556 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
703
+ 2024-04-26 16:07:36,944 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
704
+ 2024-04-26 16:07:38,945 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
705
+ 2024-04-26 16:07:40,804 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
706
+ 2024-04-26 16:07:40,948 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
707
+ 2024-04-26 16:07:42,949 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
708
+ 2024-04-26 16:07:44,951 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
709
+ 2024-04-26 16:07:46,213 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
710
+ 2024-04-26 16:07:46,286 DEBUG SenderThread:713738 [sender.py:send():379] send: stats
711
+ 2024-04-26 16:07:46,953 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
712
+ 2024-04-26 16:07:48,956 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
713
+ 2024-04-26 16:07:50,957 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
714
+ 2024-04-26 16:07:51,555 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
715
+ 2024-04-26 16:07:51,555 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
716
+ 2024-04-26 16:07:51,555 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
717
+ 2024-04-26 16:07:51,556 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
718
+ 2024-04-26 16:07:52,960 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
719
+ 2024-04-26 16:07:54,736 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
720
+ 2024-04-26 16:07:54,737 DEBUG SenderThread:713738 [sender.py:send():379] send: history
721
+ 2024-04-26 16:07:54,738 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
722
+ 2024-04-26 16:07:54,740 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
723
+ 2024-04-26 16:07:54,962 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
724
+ 2024-04-26 16:07:54,963 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
725
+ 2024-04-26 16:07:56,576 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
726
+ 2024-04-26 16:07:56,964 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
727
+ 2024-04-26 16:07:58,967 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
728
+ 2024-04-26 16:08:00,968 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
729
+ 2024-04-26 16:08:01,802 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
730
+ 2024-04-26 16:08:02,971 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
731
+ 2024-04-26 16:08:03,478 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
732
+ 2024-04-26 16:08:03,480 DEBUG SenderThread:713738 [sender.py:send():379] send: history
733
+ 2024-04-26 16:08:03,481 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
734
+ 2024-04-26 16:08:03,483 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
735
+ 2024-04-26 16:08:03,972 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
736
+ 2024-04-26 16:08:04,973 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
737
+ 2024-04-26 16:08:06,598 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
738
+ 2024-04-26 16:08:06,599 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
739
+ 2024-04-26 16:08:06,599 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
740
+ 2024-04-26 16:08:07,726 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
741
+ 2024-04-26 16:08:12,727 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
742
+ 2024-04-26 16:08:14,985 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
743
+ 2024-04-26 16:08:16,275 DEBUG SenderThread:713738 [sender.py:send():379] send: stats
744
+ 2024-04-26 16:08:18,276 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
745
+ 2024-04-26 16:08:22,386 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
746
+ 2024-04-26 16:08:23,064 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
747
+ 2024-04-26 16:08:23,064 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
748
+ 2024-04-26 16:08:24,187 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
749
+ 2024-04-26 16:08:29,188 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240426_155715-vljs8xzo/files/output.log CHANGED
@@ -241,3 +241,88 @@
241
  {'eval_loss': 0.6438009142875671, 'eval_runtime': 8.3559, 'eval_samples_per_second': 239.351, 'eval_steps_per_second': 3.83, 'eval_rewards/chosen': -0.10771973431110382, 'eval_rewards/rejected': -0.24101632833480835, 'eval_rewards/accuracies': 0.62109375, 'eval_rewards/margins': 0.13329659402370453, 'eval_logps/rejected': -428.6236572265625, 'eval_logps/chosen': -433.9639892578125, 'eval_logits/rejected': -3.2049574851989746, 'eval_logits/chosen': -3.2553329467773438, 'epoch': 0.63}
242
  [INFO|modeling_utils.py:2598] 2024-04-26 16:05:20,618 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 2 checkpoint shards. You can find where each parameters has been saved in the index located at ./checkpoint-300/model.safetensors.index.json.
243
  [INFO|tokenization_utils_base.py:2488] 2024-04-26 16:05:20,621 >> tokenizer config file saved in ./checkpoint-300/tokenizer_config.json
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
241
  {'eval_loss': 0.6438009142875671, 'eval_runtime': 8.3559, 'eval_samples_per_second': 239.351, 'eval_steps_per_second': 3.83, 'eval_rewards/chosen': -0.10771973431110382, 'eval_rewards/rejected': -0.24101632833480835, 'eval_rewards/accuracies': 0.62109375, 'eval_rewards/margins': 0.13329659402370453, 'eval_logps/rejected': -428.6236572265625, 'eval_logps/chosen': -433.9639892578125, 'eval_logits/rejected': -3.2049574851989746, 'eval_logits/chosen': -3.2553329467773438, 'epoch': 0.63}
242
  [INFO|modeling_utils.py:2598] 2024-04-26 16:05:20,618 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 2 checkpoint shards. You can find where each parameters has been saved in the index located at ./checkpoint-300/model.safetensors.index.json.
243
  [INFO|tokenization_utils_base.py:2488] 2024-04-26 16:05:20,621 >> tokenizer config file saved in ./checkpoint-300/tokenizer_config.json
244
+ [INFO|tokenization_utils_base.py:2497] 2024-04-26 16:05:20,623 >> Special tokens file saved in ./checkpoint-300/special_tokens_map.json
245
+ [INFO|tokenization_utils_base.py:2488] 2024-04-26 16:05:39,112 >> tokenizer config file saved in ./tokenizer_config.json
246
+ [INFO|tokenization_utils_base.py:2497] 2024-04-26 16:05:39,114 >> Special tokens file saved in ./special_tokens_map.json
247
+ [INFO|trainer.py:3397] 2024-04-26 16:05:39,143 >> Deleting older checkpoint [checkpoint-200] due to args.save_total_limit
248
+
249
+
250
+
251
+
252
+
253
+
254
+
255
+
256
+
257
+
258
+
259
+
260
+
261
+
262
+
263
+
264
+ 68%|██████▊ | 325/478 [08:53<03:20, 1.31s/it]
265
+
266
+
267
+
268
+
269
+
270
+
271
+
272
+
273
+
274
+
275
+
276
+
277
+
278
+
279
+
280
+
281
+
282
+ 73%|███████▎ | 351/478 [09:27<02:47, 1.32s/it]
283
+
284
+
285
+
286
+
287
+
288
+
289
+
290
+
291
+
292
+
293
+
294
+
295
+
296
+
297
+
298
+
299
+
300
+ 79%|███████▊ | 376/478 [10:01<02:20, 1.37s/it]
301
+
302
+
303
+
304
+
305
+
306
+
307
+
308
+
309
+
310
+
311
+
312
+
313
+
314
+
315
+
316
+
317
+ 84%|████████▎ | 400/478 [10:34<01:45, 1.35s/it][INFO|trainer.py:3614] 2024-04-26 16:07:54,737 >> ***** Running Evaluation *****
318
+ [INFO|trainer.py:3616] 2024-04-26 16:07:54,737 >> Num examples = 2000
319
+ [INFO|trainer.py:3619] 2024-04-26 16:07:54,737 >> Batch size = 8
320
+ 12%|█▎ | 4/32 [00:00<00:07, 3.96it/s]
321
+
322
+
323
+
324
+ [INFO|configuration_utils.py:471] 2024-04-26 16:08:03,486 >> Configuration saved in ./checkpoint-400/config.json
325
+ [INFO|configuration_utils.py:697] 2024-04-26 16:08:03,490 >> Configuration saved in ./checkpoint-400/generation_config.json
326
+ {'eval_loss': 0.6415477395057678, 'eval_runtime': 8.7287, 'eval_samples_per_second': 229.13, 'eval_steps_per_second': 3.666, 'eval_rewards/chosen': -0.10007989406585693, 'eval_rewards/rejected': -0.24366310238838196, 'eval_rewards/accuracies': 0.62109375, 'eval_rewards/margins': 0.14358317852020264, 'eval_logps/rejected': -428.88836669921875, 'eval_logps/chosen': -433.20001220703125, 'eval_logits/rejected': -3.204622507095337, 'eval_logits/chosen': -3.254263401031494, 'epoch': 0.84}
327
+ [INFO|modeling_utils.py:2598] 2024-04-26 16:08:13,074 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 2 checkpoint shards. You can find where each parameters has been saved in the index located at ./checkpoint-400/model.safetensors.index.json.
328
+ [INFO|tokenization_utils_base.py:2488] 2024-04-26 16:08:13,077 >> tokenizer config file saved in ./checkpoint-400/tokenizer_config.json
wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/loss": 0.6494, "train/grad_norm": 10.980072975158691, "train/learning_rate": 1.8323929841460178e-07, "train/rewards/chosen": -0.1245601624250412, "train/rewards/rejected": -0.2303762584924698, "train/rewards/accuracies": 0.625, "train/rewards/margins": 0.10581608861684799, "train/logps/rejected": -436.2774658203125, "train/logps/chosen": -443.0799865722656, "train/logits/rejected": -3.1383109092712402, "train/logits/chosen": -3.2006824016571045, "train/epoch": 0.6276150627615062, "train/global_step": 300, "_timestamp": 1714147511.1377053, "_runtime": 475.4368362426758, "_step": 15, "eval/loss": 0.6438009142875671, "eval/runtime": 8.3559, "eval/samples_per_second": 239.351, "eval/steps_per_second": 3.83, "eval/rewards/chosen": -0.10771973431110382, "eval/rewards/rejected": -0.24101632833480835, "eval/rewards/accuracies": 0.62109375, "eval/rewards/margins": 0.13329659402370453, "eval/logps/rejected": -428.6236572265625, "eval/logps/chosen": -433.9639892578125, "eval/logits/rejected": -3.2049574851989746, "eval/logits/chosen": -3.2553329467773438}
 
1
+ {"train/loss": 0.6428, "train/grad_norm": 8.851872444152832, "train/learning_rate": 3.9507259776993954e-08, "train/rewards/chosen": -0.08884063363075256, "train/rewards/rejected": -0.22824116051197052, "train/rewards/accuracies": 0.6524999737739563, "train/rewards/margins": 0.13940051198005676, "train/logps/rejected": -409.0178527832031, "train/logps/chosen": -449.8016052246094, "train/logits/rejected": -3.1744155883789062, "train/logits/chosen": -3.221069097518921, "train/epoch": 0.8368200836820083, "train/global_step": 400, "_timestamp": 1714147683.4777088, "_runtime": 647.7768397331238, "_step": 20, "eval/loss": 0.6415477395057678, "eval/runtime": 8.7287, "eval/samples_per_second": 229.13, "eval/steps_per_second": 3.666, "eval/rewards/chosen": -0.10007989406585693, "eval/rewards/rejected": -0.24366310238838196, "eval/rewards/accuracies": 0.62109375, "eval/rewards/margins": 0.14358317852020264, "eval/logps/rejected": -428.88836669921875, "eval/logps/chosen": -433.20001220703125, "eval/logits/rejected": -3.204622507095337, "eval/logits/chosen": -3.254263401031494}
wandb/run-20240426_155715-vljs8xzo/logs/debug-internal.log CHANGED
@@ -577,3 +577,173 @@
577
  2024-04-26 16:05:37,184 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
578
  2024-04-26 16:05:37,526 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
579
  2024-04-26 16:05:38,277 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
577
  2024-04-26 16:05:37,184 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
578
  2024-04-26 16:05:37,526 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
579
  2024-04-26 16:05:38,277 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
580
+ 2024-04-26 16:05:40,699 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
581
+ 2024-04-26 16:05:42,701 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
582
+ 2024-04-26 16:05:43,378 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
583
+ 2024-04-26 16:05:44,703 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
584
+ 2024-04-26 16:05:46,265 DEBUG SenderThread:713738 [sender.py:send():379] send: stats
585
+ 2024-04-26 16:05:46,705 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
586
+ 2024-04-26 16:05:48,641 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
587
+ 2024-04-26 16:05:48,707 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
588
+ 2024-04-26 16:05:50,709 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
589
+ 2024-04-26 16:05:51,510 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
590
+ 2024-04-26 16:05:51,518 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
591
+ 2024-04-26 16:05:51,518 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
592
+ 2024-04-26 16:05:52,711 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
593
+ 2024-04-26 16:05:53,838 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
594
+ 2024-04-26 16:05:54,713 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
595
+ 2024-04-26 16:05:56,715 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
596
+ 2024-04-26 16:05:58,717 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
597
+ 2024-04-26 16:05:59,141 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
598
+ 2024-04-26 16:06:00,719 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
599
+ 2024-04-26 16:06:02,721 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
600
+ 2024-04-26 16:06:04,336 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
601
+ 2024-04-26 16:06:04,722 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
602
+ 2024-04-26 16:06:06,510 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
603
+ 2024-04-26 16:06:06,518 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
604
+ 2024-04-26 16:06:06,518 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
605
+ 2024-04-26 16:06:06,725 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
606
+ 2024-04-26 16:06:08,727 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
607
+ 2024-04-26 16:06:09,626 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
608
+ 2024-04-26 16:06:10,729 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
609
+ 2024-04-26 16:06:12,730 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
610
+ 2024-04-26 16:06:13,597 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
611
+ 2024-04-26 16:06:13,599 DEBUG SenderThread:713738 [sender.py:send():379] send: history
612
+ 2024-04-26 16:06:13,599 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
613
+ 2024-04-26 16:06:13,601 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
614
+ 2024-04-26 16:06:13,732 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
615
+ 2024-04-26 16:06:14,733 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
616
+ 2024-04-26 16:06:14,912 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
617
+ 2024-04-26 16:06:16,267 DEBUG SenderThread:713738 [sender.py:send():379] send: stats
618
+ 2024-04-26 16:06:16,735 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
619
+ 2024-04-26 16:06:18,737 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
620
+ 2024-04-26 16:06:20,181 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
621
+ 2024-04-26 16:06:20,739 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
622
+ 2024-04-26 16:06:21,510 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
623
+ 2024-04-26 16:06:21,518 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
624
+ 2024-04-26 16:06:21,518 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
625
+ 2024-04-26 16:06:22,741 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
626
+ 2024-04-26 16:06:24,743 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
627
+ 2024-04-26 16:06:25,488 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
628
+ 2024-04-26 16:06:26,745 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
629
+ 2024-04-26 16:06:28,747 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
630
+ 2024-04-26 16:06:30,678 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
631
+ 2024-04-26 16:06:30,749 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
632
+ 2024-04-26 16:06:32,751 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
633
+ 2024-04-26 16:06:34,753 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
634
+ 2024-04-26 16:06:35,814 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
635
+ 2024-04-26 16:06:36,520 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
636
+ 2024-04-26 16:06:36,520 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
637
+ 2024-04-26 16:06:36,521 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
638
+ 2024-04-26 16:06:36,755 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
639
+ 2024-04-26 16:06:38,757 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
640
+ 2024-04-26 16:06:40,759 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
641
+ 2024-04-26 16:06:41,043 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
642
+ 2024-04-26 16:06:42,760 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
643
+ 2024-04-26 16:06:44,762 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
644
+ 2024-04-26 16:06:46,257 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
645
+ 2024-04-26 16:06:46,268 DEBUG SenderThread:713738 [sender.py:send():379] send: stats
646
+ 2024-04-26 16:06:46,317 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
647
+ 2024-04-26 16:06:46,318 DEBUG SenderThread:713738 [sender.py:send():379] send: history
648
+ 2024-04-26 16:06:46,319 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
649
+ 2024-04-26 16:06:46,320 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
650
+ 2024-04-26 16:06:46,765 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
651
+ 2024-04-26 16:06:46,765 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
652
+ 2024-04-26 16:06:48,767 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
653
+ 2024-04-26 16:06:50,769 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
654
+ 2024-04-26 16:06:51,468 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
655
+ 2024-04-26 16:06:51,555 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
656
+ 2024-04-26 16:06:51,555 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
657
+ 2024-04-26 16:06:51,557 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
658
+ 2024-04-26 16:06:52,771 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
659
+ 2024-04-26 16:06:54,866 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
660
+ 2024-04-26 16:06:56,733 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
661
+ 2024-04-26 16:06:56,868 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
662
+ 2024-04-26 16:06:58,895 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
663
+ 2024-04-26 16:07:00,897 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
664
+ 2024-04-26 16:07:02,561 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
665
+ 2024-04-26 16:07:02,899 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
666
+ 2024-04-26 16:07:04,900 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
667
+ 2024-04-26 16:07:06,554 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
668
+ 2024-04-26 16:07:06,555 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
669
+ 2024-04-26 16:07:06,555 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
670
+ 2024-04-26 16:07:06,904 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
671
+ 2024-04-26 16:07:07,900 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
672
+ 2024-04-26 16:07:08,907 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
673
+ 2024-04-26 16:07:10,910 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
674
+ 2024-04-26 16:07:12,913 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
675
+ 2024-04-26 16:07:13,514 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
676
+ 2024-04-26 16:07:14,916 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
677
+ 2024-04-26 16:07:16,271 DEBUG SenderThread:713738 [sender.py:send():379] send: stats
678
+ 2024-04-26 16:07:16,920 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
679
+ 2024-04-26 16:07:18,908 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
680
+ 2024-04-26 16:07:18,924 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
681
+ 2024-04-26 16:07:20,750 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
682
+ 2024-04-26 16:07:20,752 DEBUG SenderThread:713738 [sender.py:send():379] send: history
683
+ 2024-04-26 16:07:20,753 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
684
+ 2024-04-26 16:07:20,757 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
685
+ 2024-04-26 16:07:20,928 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
686
+ 2024-04-26 16:07:20,928 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
687
+ 2024-04-26 16:07:21,555 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
688
+ 2024-04-26 16:07:21,555 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
689
+ 2024-04-26 16:07:21,556 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
690
+ 2024-04-26 16:07:22,930 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
691
+ 2024-04-26 16:07:24,343 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
692
+ 2024-04-26 16:07:24,932 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
693
+ 2024-04-26 16:07:26,933 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
694
+ 2024-04-26 16:07:28,935 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
695
+ 2024-04-26 16:07:30,082 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
696
+ 2024-04-26 16:07:30,937 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
697
+ 2024-04-26 16:07:32,939 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
698
+ 2024-04-26 16:07:34,941 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
699
+ 2024-04-26 16:07:35,448 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
700
+ 2024-04-26 16:07:36,555 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
701
+ 2024-04-26 16:07:36,555 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
702
+ 2024-04-26 16:07:36,556 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
703
+ 2024-04-26 16:07:36,944 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
704
+ 2024-04-26 16:07:38,945 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
705
+ 2024-04-26 16:07:40,804 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
706
+ 2024-04-26 16:07:40,948 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
707
+ 2024-04-26 16:07:42,949 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
708
+ 2024-04-26 16:07:44,951 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
709
+ 2024-04-26 16:07:46,213 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
710
+ 2024-04-26 16:07:46,286 DEBUG SenderThread:713738 [sender.py:send():379] send: stats
711
+ 2024-04-26 16:07:46,953 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
712
+ 2024-04-26 16:07:48,956 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
713
+ 2024-04-26 16:07:50,957 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
714
+ 2024-04-26 16:07:51,555 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
715
+ 2024-04-26 16:07:51,555 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
716
+ 2024-04-26 16:07:51,555 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
717
+ 2024-04-26 16:07:51,556 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
718
+ 2024-04-26 16:07:52,960 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
719
+ 2024-04-26 16:07:54,736 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
720
+ 2024-04-26 16:07:54,737 DEBUG SenderThread:713738 [sender.py:send():379] send: history
721
+ 2024-04-26 16:07:54,738 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
722
+ 2024-04-26 16:07:54,740 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
723
+ 2024-04-26 16:07:54,962 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
724
+ 2024-04-26 16:07:54,963 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
725
+ 2024-04-26 16:07:56,576 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
726
+ 2024-04-26 16:07:56,964 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
727
+ 2024-04-26 16:07:58,967 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
728
+ 2024-04-26 16:08:00,968 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
729
+ 2024-04-26 16:08:01,802 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
730
+ 2024-04-26 16:08:02,971 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
731
+ 2024-04-26 16:08:03,478 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: partial_history
732
+ 2024-04-26 16:08:03,480 DEBUG SenderThread:713738 [sender.py:send():379] send: history
733
+ 2024-04-26 16:08:03,481 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: summary_record
734
+ 2024-04-26 16:08:03,483 INFO SenderThread:713738 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
735
+ 2024-04-26 16:08:03,972 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/wandb-summary.json
736
+ 2024-04-26 16:08:04,973 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
737
+ 2024-04-26 16:08:06,598 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
738
+ 2024-04-26 16:08:06,599 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
739
+ 2024-04-26 16:08:06,599 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
740
+ 2024-04-26 16:08:07,726 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
741
+ 2024-04-26 16:08:12,727 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
742
+ 2024-04-26 16:08:14,985 INFO Thread-12 :713738 [dir_watcher.py:_on_file_modified():288] file/dir modified: /fsx/sanchit/distil-zephyr-1.5b-dpo-ultrafeedback/wandb/run-20240426_155715-vljs8xzo/files/output.log
743
+ 2024-04-26 16:08:16,275 DEBUG SenderThread:713738 [sender.py:send():379] send: stats
744
+ 2024-04-26 16:08:18,276 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
745
+ 2024-04-26 16:08:22,386 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: internal_messages
746
+ 2024-04-26 16:08:23,064 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: stop_status
747
+ 2024-04-26 16:08:23,064 DEBUG SenderThread:713738 [sender.py:send_request():406] send_request: stop_status
748
+ 2024-04-26 16:08:24,187 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
749
+ 2024-04-26 16:08:29,188 DEBUG HandlerThread:713738 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240426_155715-vljs8xzo/run-vljs8xzo.wandb CHANGED
Binary files a/wandb/run-20240426_155715-vljs8xzo/run-vljs8xzo.wandb and b/wandb/run-20240426_155715-vljs8xzo/run-vljs8xzo.wandb differ