2024-05-05 21:36:24,243 INFO    StreamThr :1215779 [internal.py:wandb_internal():88] W&B internal server running at pid: 1215779, started at: 2024-05-05 21:36:24.243188
2024-05-05 21:36:24,245 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: status
2024-05-05 21:36:24,269 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: status
2024-05-05 21:36:24,270 DEBUG   SenderThread:1215779 [tracking_sender.py:send():315] send: header
2024-05-05 21:36:24,270 INFO    WriterThread:1215779 [datastore.py:open_for_write():77] open: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/run-mvsl6gev.wandb
2024-05-05 21:36:24,270 DEBUG   SenderThread:1215779 [tracking_sender.py:send():315] send: run
2024-05-05 21:36:24,272 INFO    SenderThread:1215779 [tracking_sender.py:_maybe_setup_resume():588] checking resume status for None/verifier-reproduce/mvsl6gev
2024-05-05 21:36:25,070 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: check_version
2024-05-05 21:36:25,073 INFO    SenderThread:1215779 [dir_watcher.py:__init__():216] watching files in: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files
2024-05-05 21:36:25,073 INFO    SenderThread:1215779 [tracking_sender.py:_start_run_threads():934] run started: mvsl6gev with start time 1714916184.241288
2024-05-05 21:36:25,073 DEBUG   SenderThread:1215779 [tracking_sender.py:send():315] send: summary
2024-05-05 21:36:25,073 INFO    SenderThread:1215779 [tracking_sender.py:_save_file():1177] saving file wandb-summary.json with policy end
2024-05-05 21:36:25,073 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: check_version
2024-05-05 21:36:26,074 INFO    Thread-15 :1215779 [dir_watcher.py:_on_file_created():275] file/dir created: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/wandb-summary.json
2024-05-05 21:36:30,073 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: run_start
2024-05-05 21:36:30,113 DEBUG   HandlerThread:1215779 [system_info.py:__init__():31] System info init
2024-05-05 21:36:30,113 DEBUG   HandlerThread:1215779 [system_info.py:__init__():46] System info init done
2024-05-05 21:36:30,113 INFO    HandlerThread:1215779 [system_monitor.py:start():150] Starting system monitor
2024-05-05 21:36:30,113 INFO    SystemMonitor:1215779 [system_monitor.py:_start():116] Starting system asset monitoring threads
2024-05-05 21:36:30,113 INFO    HandlerThread:1215779 [system_monitor.py:probe():168] Collecting system info
2024-05-05 21:36:30,114 INFO    SystemMonitor:1215779 [interfaces.py:start():168] Started cpu
2024-05-05 21:36:30,114 INFO    SystemMonitor:1215779 [interfaces.py:start():168] Started disk
2024-05-05 21:36:30,114 INFO    SystemMonitor:1215779 [interfaces.py:start():168] Started gpu
2024-05-05 21:36:30,115 INFO    SystemMonitor:1215779 [interfaces.py:start():168] Started memory
2024-05-05 21:36:30,116 INFO    SystemMonitor:1215779 [interfaces.py:start():168] Started network
2024-05-05 21:36:30,167 DEBUG   HandlerThread:1215779 [system_info.py:probe():195] Probing system
2024-05-05 21:36:30,174 DEBUG   HandlerThread:1215779 [git.py:repo():40] git repository is invalid
2024-05-05 21:36:30,174 DEBUG   HandlerThread:1215779 [system_info.py:probe():241] Probing system done
2024-05-05 21:36:30,174 DEBUG   HandlerThread:1215779 [system_monitor.py:probe():177] {'os': 'Linux-5.10.135.bsk.6-amd64-x86_64-with-glibc2.31', 'python': '3.9.2', 'heartbeatAt': '2024-05-05T13:36:30.167949', 'startedAt': '2024-05-05T13:36:24.236025', 'docker': None, 'cuda': None, 'args': ('--model_name_or_path', '../models/lean4_random_15k_all/checkpoint-236/', '--data_dir', '../repl/pass_rate_results/lean4_basic_test/lean4_random_15k_allpass5.jsonl', '--data_id', 'lean4test', '--target_set', 'train', '--save_dir', '../models/lean4_random_15k_all-sample5-test-gt', '--generator_id', 'lean4_random_15k_all', '--verifier_id', 'test', '--dedup', 'True', '--per_problem_sampling_solution', '5', '--loss_level', 'token', '--loss_on_llm', 'True', '--num_train_epoches', '1', '--eval_steps', '10000', '--per_device_train_batch_size', '16', '--per_device_eval_batch_size', '64', '--gradient_accumulation_steps', '4', '--gradient_checkpointing', 'True', '--learning_rate', '2e-6', '--weight_decay', '0', '--save_steps', '1000', '--lr_scheduler_type', 'linear', '--warmup_steps', '0', '--save_epoches', '1', '--save_best', 'False', '--save_total_limit', '0', '--logging_dir', 'None', '--logging_steps', '1', '--easy', 'True', '--process', 'True', '--seed', '42'), 'state': 'running', 'program': '/opt/tiger/auto-info/PSV/train_psv.py', 'codePath': 'PSV/train_psv.py', 'host': 'n122-200-048', 'username': 'tiger', 'executable': '/usr/bin/python3', 'cpu_count': 90, 'cpu_count_logical': 180, 'cpu_freq': {'current': 2599.5619999999913, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}, {'current': 2599.562, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 3519.748878479004, 'used': 522.4962768554688}, 'gpu': 'NVIDIA H800', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA H800', 'memory_total': 85520809984}, {'name': 'NVIDIA H800', 'memory_total': 85520809984}, {'name': 'NVIDIA H800', 'memory_total': 85520809984}, {'name': 'NVIDIA H800', 'memory_total': 85520809984}, {'name': 'NVIDIA H800', 'memory_total': 85520809984}, {'name': 'NVIDIA H800', 'memory_total': 85520809984}, {'name': 'NVIDIA H800', 'memory_total': 85520809984}, {'name': 'NVIDIA H800', 'memory_total': 85520809984}], 'memory': {'total': 2932.9151611328125}}
2024-05-05 21:36:30,174 INFO    HandlerThread:1215779 [system_monitor.py:probe():178] Finished collecting system info
2024-05-05 21:36:30,174 INFO    HandlerThread:1215779 [system_monitor.py:probe():181] Publishing system info
2024-05-05 21:36:30,174 DEBUG   HandlerThread:1215779 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment
2024-05-05 21:36:30,175 DEBUG   HandlerThread:1215779 [system_info.py:_save_pip():67] Saving pip packages done
2024-05-05 21:36:30,176 INFO    HandlerThread:1215779 [system_monitor.py:probe():183] Finished publishing system info
2024-05-05 21:36:30,210 DEBUG   SenderThread:1215779 [tracking_sender.py:send():315] send: files
2024-05-05 21:36:30,210 INFO    SenderThread:1215779 [tracking_sender.py:_save_file():1177] saving file wandb-metadata.json with policy now
2024-05-05 21:36:30,213 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: stop_status
2024-05-05 21:36:30,214 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: stop_status
2024-05-05 21:36:30,214 DEBUG   SenderThread:1215779 [tracking_sender.py:send():315] send: telemetry
2024-05-05 21:36:30,393 INFO    Thread-17 :1215779 [upload_job.py:push():143] Uploaded file /tmp/tmpk9y5fh7cwandb/ns8mhjsx-wandb-metadata.json
2024-05-05 21:36:31,075 INFO    Thread-15 :1215779 [dir_watcher.py:_on_file_created():275] file/dir created: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/requirements.txt
2024-05-05 21:36:31,075 INFO    Thread-15 :1215779 [dir_watcher.py:_on_file_created():275] file/dir created: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/wandb-metadata.json
2024-05-05 21:36:31,075 INFO    Thread-15 :1215779 [dir_watcher.py:_on_file_created():275] file/dir created: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/output.log
2024-05-05 21:36:33,075 INFO    Thread-15 :1215779 [dir_watcher.py:_on_file_modified():292] file/dir modified: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/output.log
2024-05-05 21:36:41,077 INFO    Thread-15 :1215779 [dir_watcher.py:_on_file_modified():292] file/dir modified: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/output.log
2024-05-05 21:36:41,611 DEBUG   SenderThread:1215779 [tracking_sender.py:send():315] send: exit
2024-05-05 21:36:41,612 INFO    SenderThread:1215779 [tracking_sender.py:send_exit():438] handling exit code: 1
2024-05-05 21:36:41,612 INFO    SenderThread:1215779 [tracking_sender.py:send_exit():440] handling runtime: 11
2024-05-05 21:36:41,612 INFO    SenderThread:1215779 [tracking_sender.py:_save_file():1177] saving file wandb-summary.json with policy end
2024-05-05 21:36:41,612 INFO    SenderThread:1215779 [tracking_sender.py:send_exit():446] send defer
2024-05-05 21:36:41,612 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: defer
2024-05-05 21:36:41,612 INFO    HandlerThread:1215779 [handler.py:handle_request_defer():163] handle defer: 0
2024-05-05 21:36:41,613 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: defer
2024-05-05 21:36:41,613 INFO    SenderThread:1215779 [tracking_sender.py:send_request_defer():455] handle sender defer: 0
2024-05-05 21:36:41,613 INFO    SenderThread:1215779 [tracking_sender.py:transition_state():459] send defer: 1
2024-05-05 21:36:41,613 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: defer
2024-05-05 21:36:41,613 INFO    HandlerThread:1215779 [handler.py:handle_request_defer():163] handle defer: 1
2024-05-05 21:36:41,613 INFO    HandlerThread:1215779 [system_monitor.py:finish():160] Stopping system monitor
2024-05-05 21:36:41,613 DEBUG   SystemMonitor:1215779 [system_monitor.py:_start():130] Starting system metrics aggregation loop
2024-05-05 21:36:41,614 DEBUG   SystemMonitor:1215779 [system_monitor.py:_start():137] Finished system metrics aggregation loop
2024-05-05 21:36:41,614 DEBUG   SystemMonitor:1215779 [system_monitor.py:_start():141] Publishing last batch of metrics
2024-05-05 21:36:41,615 INFO    HandlerThread:1215779 [interfaces.py:finish():173] Joined cpu
2024-05-05 21:36:41,615 INFO    HandlerThread:1215779 [interfaces.py:finish():173] Joined disk
2024-05-05 21:36:41,660 INFO    HandlerThread:1215779 [interfaces.py:finish():173] Joined gpu
2024-05-05 21:36:41,660 INFO    HandlerThread:1215779 [interfaces.py:finish():173] Joined memory
2024-05-05 21:36:41,660 INFO    HandlerThread:1215779 [interfaces.py:finish():173] Joined network
2024-05-05 21:36:41,660 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: defer
2024-05-05 21:36:41,660 INFO    SenderThread:1215779 [tracking_sender.py:send_request_defer():455] handle sender defer: 1
2024-05-05 21:36:41,660 INFO    SenderThread:1215779 [tracking_sender.py:transition_state():459] send defer: 2
2024-05-05 21:36:41,660 DEBUG   SenderThread:1215779 [tracking_sender.py:send():315] send: stats
2024-05-05 21:36:41,662 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: defer
2024-05-05 21:36:41,662 INFO    HandlerThread:1215779 [handler.py:handle_request_defer():163] handle defer: 2
2024-05-05 21:36:41,662 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: defer
2024-05-05 21:36:41,662 INFO    SenderThread:1215779 [tracking_sender.py:send_request_defer():455] handle sender defer: 2
2024-05-05 21:36:41,662 INFO    SenderThread:1215779 [tracking_sender.py:transition_state():459] send defer: 3
2024-05-05 21:36:41,662 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: defer
2024-05-05 21:36:41,662 INFO    HandlerThread:1215779 [handler.py:handle_request_defer():163] handle defer: 3
2024-05-05 21:36:41,662 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: defer
2024-05-05 21:36:41,662 INFO    SenderThread:1215779 [tracking_sender.py:send_request_defer():455] handle sender defer: 3
2024-05-05 21:36:41,662 INFO    SenderThread:1215779 [tracking_sender.py:transition_state():459] send defer: 4
2024-05-05 21:36:41,662 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: defer
2024-05-05 21:36:41,662 INFO    HandlerThread:1215779 [handler.py:handle_request_defer():163] handle defer: 4
2024-05-05 21:36:41,662 DEBUG   SenderThread:1215779 [tracking_sender.py:send():315] send: summary
2024-05-05 21:36:41,663 INFO    SenderThread:1215779 [tracking_sender.py:_save_file():1177] saving file wandb-summary.json with policy end
2024-05-05 21:36:41,663 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: defer
2024-05-05 21:36:41,663 INFO    SenderThread:1215779 [tracking_sender.py:send_request_defer():455] handle sender defer: 4
2024-05-05 21:36:41,663 INFO    SenderThread:1215779 [tracking_sender.py:transition_state():459] send defer: 5
2024-05-05 21:36:41,663 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: defer
2024-05-05 21:36:41,663 INFO    HandlerThread:1215779 [handler.py:handle_request_defer():163] handle defer: 5
2024-05-05 21:36:41,663 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: defer
2024-05-05 21:36:41,663 INFO    SenderThread:1215779 [tracking_sender.py:send_request_defer():455] handle sender defer: 5
2024-05-05 21:36:41,663 INFO    SenderThread:1215779 [tracking_sender.py:transition_state():459] send defer: 6
2024-05-05 21:36:41,664 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: defer
2024-05-05 21:36:41,664 INFO    HandlerThread:1215779 [handler.py:handle_request_defer():163] handle defer: 6
2024-05-05 21:36:41,664 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: defer
2024-05-05 21:36:41,664 INFO    SenderThread:1215779 [tracking_sender.py:send_request_defer():455] handle sender defer: 6
2024-05-05 21:36:42,077 INFO    Thread-15 :1215779 [dir_watcher.py:_on_file_modified():292] file/dir modified: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/config.yaml
2024-05-05 21:36:42,078 INFO    Thread-15 :1215779 [dir_watcher.py:_on_file_modified():292] file/dir modified: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/wandb-summary.json
2024-05-05 21:36:42,238 INFO    SenderThread:1215779 [tracking_sender.py:transition_state():459] send defer: 7
2024-05-05 21:36:42,238 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: defer
2024-05-05 21:36:42,238 INFO    HandlerThread:1215779 [handler.py:handle_request_defer():163] handle defer: 7
2024-05-05 21:36:42,239 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: defer
2024-05-05 21:36:42,239 INFO    SenderThread:1215779 [tracking_sender.py:send_request_defer():455] handle sender defer: 7
2024-05-05 21:36:42,239 INFO    SenderThread:1215779 [dir_watcher.py:finish():362] shutting down directory watcher
2024-05-05 21:36:42,611 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: poll_exit
2024-05-05 21:36:43,078 INFO    Thread-15 :1215779 [dir_watcher.py:_on_file_modified():292] file/dir modified: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/output.log
2024-05-05 21:36:43,078 INFO    SenderThread:1215779 [dir_watcher.py:finish():392] scan: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files
2024-05-05 21:36:43,078 INFO    SenderThread:1215779 [dir_watcher.py:finish():406] scan save: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/output.log output.log
2024-05-05 21:36:43,078 INFO    SenderThread:1215779 [dir_watcher.py:finish():406] scan save: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/config.yaml config.yaml
2024-05-05 21:36:43,079 INFO    SenderThread:1215779 [dir_watcher.py:finish():406] scan save: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/wandb-summary.json wandb-summary.json
2024-05-05 21:36:43,079 INFO    SenderThread:1215779 [dir_watcher.py:finish():406] scan save: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/requirements.txt requirements.txt
2024-05-05 21:36:43,079 INFO    SenderThread:1215779 [dir_watcher.py:finish():406] scan save: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/wandb-metadata.json wandb-metadata.json
2024-05-05 21:36:43,079 INFO    SenderThread:1215779 [tracking_sender.py:transition_state():459] send defer: 8
2024-05-05 21:36:43,079 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: poll_exit
2024-05-05 21:36:43,082 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: defer
2024-05-05 21:36:43,083 INFO    HandlerThread:1215779 [handler.py:handle_request_defer():163] handle defer: 8
2024-05-05 21:36:43,083 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: defer
2024-05-05 21:36:43,083 INFO    SenderThread:1215779 [tracking_sender.py:send_request_defer():455] handle sender defer: 8
2024-05-05 21:36:43,083 INFO    SenderThread:1215779 [file_pusher.py:finish():168] shutting down file pusher
2024-05-05 21:36:43,110 INFO    Thread-19 :1215779 [upload_job.py:push():143] Uploaded file None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/config.yaml
2024-05-05 21:36:43,113 INFO    Thread-18 :1215779 [upload_job.py:push():143] Uploaded file None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/output.log
2024-05-05 21:36:43,116 INFO    Thread-20 :1215779 [upload_job.py:push():143] Uploaded file None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/wandb-summary.json
2024-05-05 21:36:43,138 INFO    Thread-21 :1215779 [upload_job.py:push():143] Uploaded file None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/files/requirements.txt
2024-05-05 21:36:43,339 INFO    Thread-14 :1215779 [tracking_sender.py:transition_state():459] send defer: 9
2024-05-05 21:36:43,339 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: defer
2024-05-05 21:36:43,339 INFO    HandlerThread:1215779 [handler.py:handle_request_defer():163] handle defer: 9
2024-05-05 21:36:43,340 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: defer
2024-05-05 21:36:43,340 INFO    SenderThread:1215779 [tracking_sender.py:send_request_defer():455] handle sender defer: 9
2024-05-05 21:36:43,340 INFO    SenderThread:1215779 [file_pusher.py:join():173] waiting for file pusher
2024-05-05 21:36:43,340 INFO    SenderThread:1215779 [tracking_sender.py:transition_state():459] send defer: 10
2024-05-05 21:36:43,340 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: defer
2024-05-05 21:36:43,340 INFO    HandlerThread:1215779 [handler.py:handle_request_defer():163] handle defer: 10
2024-05-05 21:36:43,340 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: defer
2024-05-05 21:36:43,340 INFO    SenderThread:1215779 [tracking_sender.py:send_request_defer():455] handle sender defer: 10
2024-05-05 21:36:43,455 INFO    SenderThread:1215779 [tracking_sender.py:transition_state():459] send defer: 11
2024-05-05 21:36:43,455 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: defer
2024-05-05 21:36:43,455 INFO    HandlerThread:1215779 [handler.py:handle_request_defer():163] handle defer: 11
2024-05-05 21:36:43,455 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: defer
2024-05-05 21:36:43,455 INFO    SenderThread:1215779 [tracking_sender.py:send_request_defer():455] handle sender defer: 11
2024-05-05 21:36:43,455 INFO    SenderThread:1215779 [tracking_sender.py:transition_state():459] send defer: 12
2024-05-05 21:36:43,456 DEBUG   SenderThread:1215779 [tracking_sender.py:send():315] send: final
2024-05-05 21:36:43,456 DEBUG   SenderThread:1215779 [tracking_sender.py:send():315] send: footer
2024-05-05 21:36:43,456 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: defer
2024-05-05 21:36:43,456 INFO    HandlerThread:1215779 [handler.py:handle_request_defer():163] handle defer: 12
2024-05-05 21:36:43,456 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: defer
2024-05-05 21:36:43,456 INFO    SenderThread:1215779 [tracking_sender.py:send_request_defer():455] handle sender defer: 12
2024-05-05 21:36:43,457 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: poll_exit
2024-05-05 21:36:43,457 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: poll_exit
2024-05-05 21:36:43,457 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: poll_exit
2024-05-05 21:36:43,457 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: poll_exit
2024-05-05 21:36:43,458 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: server_info
2024-05-05 21:36:43,458 DEBUG   SenderThread:1215779 [tracking_sender.py:send_request():329] send_request: server_info
2024-05-05 21:36:43,458 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: get_summary
2024-05-05 21:36:43,458 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: sampled_history
2024-05-05 21:36:43,459 INFO    MainThread:1215779 [wandb_run.py:_footer_history_summary_info():3391] rendering history
2024-05-05 21:36:43,459 INFO    MainThread:1215779 [wandb_run.py:_footer_history_summary_info():3423] rendering summary
2024-05-05 21:36:43,459 INFO    MainThread:1215779 [wandb_run.py:_footer_sync_info():3349] logging synced files
2024-05-05 21:36:43,459 DEBUG   HandlerThread:1215779 [handler.py:handle_request():140] handle_request: shutdown
2024-05-05 21:36:43,459 INFO    HandlerThread:1215779 [handler.py:finish():817] shutting down handler
2024-05-05 21:36:44,456 INFO    WriterThread:1215779 [datastore.py:close():283] close: None/verifier-reproduce/wandb/run-20240505_213624-mvsl6gev/run-mvsl6gev.wandb
2024-05-05 21:36:44,458 INFO    SenderThread:1215779 [tracking_sender.py:finish():1338] shutting down sender
2024-05-05 21:36:44,458 INFO    SenderThread:1215779 [file_pusher.py:finish():168] shutting down file pusher
2024-05-05 21:36:44,458 INFO    SenderThread:1215779 [file_pusher.py:join():173] waiting for file pusher
2024-05-05 21:36:44,461 INFO    MainThread:1215779 [internal.py:handle_exit():78] Internal process exited