2025-01-04 20:33:14,208 INFO StreamThr :2321186 [internal.py:wandb_internal():85] W&B internal server running at pid: 2321186, started at: 2025-01-04 20:33:14.206914 2025-01-04 20:33:14,211 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status 2025-01-04 20:33:14,219 INFO WriterThread:2321186 [datastore.py:open_for_write():87] open: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/run-95wq5z4x.wandb 2025-01-04 20:33:14,220 DEBUG SenderThread:2321186 [sender.py:send():379] send: header 2025-01-04 20:33:14,298 DEBUG SenderThread:2321186 [sender.py:send():379] send: run 2025-01-04 20:33:14,524 INFO SenderThread:2321186 [dir_watcher.py:__init__():211] watching files in: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files 2025-01-04 20:33:14,524 INFO SenderThread:2321186 [sender.py:_start_run_threads():1188] run started: 95wq5z4x with start time 1736022794.21094 2025-01-04 20:33:14,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: check_version 2025-01-04 20:33:14,548 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: check_version 2025-01-04 20:33:14,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: run_start 2025-01-04 20:33:14,650 DEBUG HandlerThread:2321186 [system_info.py:__init__():26] System info init 2025-01-04 20:33:14,650 DEBUG HandlerThread:2321186 [system_info.py:__init__():41] System info init done 2025-01-04 20:33:14,651 INFO HandlerThread:2321186 [system_monitor.py:start():194] Starting system monitor 2025-01-04 20:33:14,651 INFO SystemMonitor:2321186 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-01-04 20:33:14,651 INFO HandlerThread:2321186 [system_monitor.py:probe():214] Collecting system info 2025-01-04 20:33:14,652 INFO SystemMonitor:2321186 [interfaces.py:start():188] Started cpu monitoring 2025-01-04 20:33:14,653 INFO SystemMonitor:2321186 [interfaces.py:start():188] Started disk monitoring 2025-01-04 20:33:14,654 INFO SystemMonitor:2321186 [interfaces.py:start():188] Started gpu monitoring 2025-01-04 20:33:14,659 INFO SystemMonitor:2321186 [interfaces.py:start():188] Started memory monitoring 2025-01-04 20:33:14,660 INFO SystemMonitor:2321186 [interfaces.py:start():188] Started network monitoring 2025-01-04 20:33:14,710 DEBUG HandlerThread:2321186 [system_info.py:probe():152] Probing system 2025-01-04 20:33:14,714 DEBUG HandlerThread:2321186 [system_info.py:_probe_git():137] Probing git 2025-01-04 20:33:14,750 DEBUG HandlerThread:2321186 [system_info.py:_probe_git():145] Probing git done 2025-01-04 20:33:14,750 DEBUG HandlerThread:2321186 [system_info.py:probe():200] Probing system done 2025-01-04 20:33:14,750 DEBUG HandlerThread:2321186 [system_monitor.py:probe():223] {'os': 'Linux-6.8.0-48-generic-x86_64-with-glibc2.39', 'python': '3.10.0', 'heartbeatAt': '2025-01-04T20:33:14.710667', 'startedAt': '2025-01-04T20:33:14.192552', 'docker': None, 'cuda': None, 'args': ('--model_family', 'llama', '--apply_instruct_masks', '--token_scaled_loss', '--seq_parallel_size', '4', '--report_to', 'wandb', '--do_train', '--model_name_or_path', '/scratch3/workspace/ctpham_umass_edu-ft/models/Llama-3-8B-ProLong-512k-Base/', '--config_name', '/scratch3/workspace/ctpham_umass_edu-ft/models/Llama-3-8B-ProLong-512k-Base/', '--tokenizer_name', '/scratch3/workspace/ctpham_umass_edu-ft/models/Llama-3-8B-ProLong-512k-Base/', '--run_name', 'wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_', '--output_dir', '/scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_', '--config_overrides_json', '', '--gradient_accumulation_steps', '4', '--per_device_train_batch_size', '1', '--bf16', '--learning_rate', '1e-5', '--min_lr_ratio', '0.1', '--lr_scheduler_type', 'cosine', '--max_grad_norm', '1.0', '--adam_beta1', '0.9', '--adam_beta2', '0.95', '--weight_decay', '0.1', '--warmup_ratio', '0.05', '--optim', 'adamw_torch', '--logging_steps', '1', '--log_level', 'info', '--save_steps', '100', '--dataloader_num_workers', '1', '--disable_tqdm', 'true', '--use_fast_tokenizer', 'false', '--remove_unused_columns', 'false', '--ddp_find_unused_parameters', 'false', '--fsdp', 'auto_wrap offload', '--gradient_checkpointing', '--tokenized_mds_train', '/work/pi_miyyer_umass_edu/ctpham/BookClaim-dev/data/ft/bookclaim_wp_pack_complete', '--cuda_empty_cache', '--num_train_epochs', '1'), 'state': 'running', 'program': '/work/pi_miyyer_umass_edu/ctpham/BookClaim-dev/prolong-final/finetune.py', 'codePathLocal': 'finetune.py', 'codePath': 'prolong-final/finetune.py', 'git': {'remote': 'https://github.com/chtmp223/BookGen-dev.git', 'commit': '9c36a7f300815c07d8a1bbceedfa1ef3781014f6'}, 'email': 'ctpham@umass.edu', 'root': '/work/pi_miyyer_umass_edu/ctpham/BookClaim-dev', 'host': 'gpu016', 'username': 'ctpham_umass_edu', 'executable': '/scratch3/workspace/ctpham_umass_edu-ft/envs/prolong-final/bin/python3.10', 'cpu_count': 112, 'cpu_count_logical': 112, 'cpu_freq': {'current': 1279.2346785714285, 'min': 800.0, 'max': 3800.0}, 'cpu_freq_per_core': [{'current': 765.676, 'min': 800.0, 'max': 3800.0}, {'current': 763.953, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 3000.0, 'min': 800.0, 'max': 3800.0}, {'current': 3000.0, 'min': 800.0, 'max': 3800.0}, {'current': 2999.969, 'min': 800.0, 'max': 3800.0}, {'current': 3000.0, 'min': 800.0, 'max': 3800.0}, {'current': 786.059, 'min': 800.0, 'max': 3800.0}, {'current': 3000.44, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 3000.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1500.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2017.085, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2400.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1400.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2074.358, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1899.98, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2400.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2059.821, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 784.691, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 793.56, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 3800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 793.216, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 3800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 756.065, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 3000.0, 'min': 800.0, 'max': 3800.0}, {'current': 785.354, 'min': 800.0, 'max': 3800.0}, {'current': 3000.0, 'min': 800.0, 'max': 3800.0}, {'current': 791.656, 'min': 800.0, 'max': 3800.0}, {'current': 3000.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2999.969, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 3000.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 3000.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 3000.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 3000.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 3000.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 3000.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 3000.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 765.39, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}], 'disk': {'/': {'total': 438.487850189209, 'used': 16.89361572265625}}, 'gpu': 'NVIDIA A100-SXM4-80GB', 'gpu_count': 4, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}], 'memory': {'total': 2015.3284606933594}} 2025-01-04 20:33:14,751 INFO HandlerThread:2321186 [system_monitor.py:probe():224] Finished collecting system info 2025-01-04 20:33:14,751 INFO HandlerThread:2321186 [system_monitor.py:probe():227] Publishing system info 2025-01-04 20:33:14,751 DEBUG HandlerThread:2321186 [system_info.py:_save_conda():209] Saving list of conda packages installed into the current environment 2025-01-04 20:33:15,526 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/conda-environment.yaml 2025-01-04 20:33:21,830 DEBUG HandlerThread:2321186 [system_info.py:_save_conda():224] Saving conda packages done 2025-01-04 20:33:21,834 INFO HandlerThread:2321186 [system_monitor.py:probe():229] Finished publishing system info 2025-01-04 20:33:21,842 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:33:21,842 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 20:33:21,843 DEBUG SenderThread:2321186 [sender.py:send():379] send: files 2025-01-04 20:33:21,844 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2025-01-04 20:33:22,178 INFO wandb-upload_0:2321186 [upload_job.py:push():130] Uploaded file /tmp/tmp8xuo15o7wandb/9vq2wa62-wandb-metadata.json 2025-01-04 20:33:22,225 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: python_packages 2025-01-04 20:33:22,226 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: python_packages 2025-01-04 20:33:22,226 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:33:22,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:22,228 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:33:22,387 DEBUG SenderThread:2321186 [sender.py:send():379] send: telemetry 2025-01-04 20:33:22,387 DEBUG SenderThread:2321186 [sender.py:send():379] send: config 2025-01-04 20:33:22,389 DEBUG SenderThread:2321186 [sender.py:send():379] send: telemetry 2025-01-04 20:33:22,389 DEBUG SenderThread:2321186 [sender.py:send():379] send: metric 2025-01-04 20:33:22,390 DEBUG SenderThread:2321186 [sender.py:send():379] send: telemetry 2025-01-04 20:33:22,390 DEBUG SenderThread:2321186 [sender.py:send():379] send: metric 2025-01-04 20:33:22,390 WARNING SenderThread:2321186 [sender.py:send_metric():1405] Seen metric with glob (shouldn't happen) 2025-01-04 20:33:22,390 DEBUG SenderThread:2321186 [sender.py:send():379] send: telemetry 2025-01-04 20:33:22,390 DEBUG SenderThread:2321186 [sender.py:send():379] send: telemetry 2025-01-04 20:33:22,390 DEBUG SenderThread:2321186 [sender.py:send():379] send: config 2025-01-04 20:33:22,528 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/conda-environment.yaml 2025-01-04 20:33:22,528 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-metadata.json 2025-01-04 20:33:22,529 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/requirements.txt 2025-01-04 20:33:23,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:24,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:25,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:25,391 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:33:26,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:27,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:28,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:29,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:30,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:31,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:33:31,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:32,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:33,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:34,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:35,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:36,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:37,226 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:33:37,227 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:33:37,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:37,393 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:33:38,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:39,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:40,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:41,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:42,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:42,541 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:33:43,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:33:43,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:44,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:44,543 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:33:45,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:46,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:47,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:48,234 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:33:48,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:48,545 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/config.yaml 2025-01-04 20:33:49,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:50,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:51,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:52,226 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:33:52,227 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:33:52,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:53,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:54,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:54,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:33:55,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:56,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:57,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:57,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:33:57,688 DEBUG SenderThread:2321186 [sender.py:send():379] send: metric 2025-01-04 20:33:57,688 DEBUG SenderThread:2321186 [sender.py:send():379] send: metric 2025-01-04 20:33:57,688 DEBUG SenderThread:2321186 [sender.py:send():379] send: metric 2025-01-04 20:33:57,688 DEBUG SenderThread:2321186 [sender.py:send():379] send: metric 2025-01-04 20:33:57,688 DEBUG SenderThread:2321186 [sender.py:send():379] send: metric 2025-01-04 20:33:57,688 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:33:57,688 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:33:57,693 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:33:58,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:58,558 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:33:59,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:33:59,726 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:34:00,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:00,561 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:34:01,232 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:02,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:03,232 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:04,232 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:05,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:34:05,232 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:06,232 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:07,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:34:07,227 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:34:07,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:08,232 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:09,233 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:10,232 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:10,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:34:11,233 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:12,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:34:12,079 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:34:12,079 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:34:12,084 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:34:12,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:12,572 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:34:13,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:14,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:14,574 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:34:14,660 DEBUG SystemMonitor:2321186 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-01-04 20:34:14,663 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:34:15,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:15,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:34:16,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:17,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:18,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:19,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:20,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:21,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:21,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:34:21,581 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/config.yaml 2025-01-04 20:34:22,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:34:22,228 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:34:22,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:23,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:24,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:25,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:26,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:34:26,284 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:34:26,285 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:34:26,286 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:34:26,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:26,585 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:34:27,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:27,324 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:34:28,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:28,587 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:34:29,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:30,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:31,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:32,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:32,342 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:34:33,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:34,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:35,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:36,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:37,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:34:37,227 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:34:37,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:37,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:34:38,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:39,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:40,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:40,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:34:40,608 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:34:40,608 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:34:40,610 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:34:41,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:41,597 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:34:42,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:42,598 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:34:43,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:43,343 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:34:44,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:44,665 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:34:45,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:46,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:47,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:48,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:34:48,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:49,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:50,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:51,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:52,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:34:52,228 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:34:52,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:53,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:54,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:34:54,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:54,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:34:54,546 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:34:54,547 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:34:54,551 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:34:54,608 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:34:55,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:56,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:56,609 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:34:57,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:58,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:34:59,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:34:59,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:00,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:01,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:02,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:03,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:04,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:35:04,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:05,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:06,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:07,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:35:07,228 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:35:07,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:08,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:08,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:35:08,566 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:35:08,566 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:35:08,568 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:35:08,619 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:35:09,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:09,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:35:10,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:10,620 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:35:11,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:12,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:13,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:14,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:14,668 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:35:14,670 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:35:15,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:16,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:17,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:18,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:19,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:20,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:35:20,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:21,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:22,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:35:22,228 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:35:22,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:22,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:35:22,860 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:35:22,861 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:35:22,866 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:35:23,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:23,630 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:35:24,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:24,631 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:35:25,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:25,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:35:26,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:27,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:28,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:29,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:30,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:31,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:35:31,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:32,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:33,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:34,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:35,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:36,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:37,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:35:37,014 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:35:37,014 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:35:37,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:35:37,016 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:35:37,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:35:37,294 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:35:37,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:37,641 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:35:38,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:38,641 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:35:39,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:40,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:41,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:42,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:35:42,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:43,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:44,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:44,671 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:35:45,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:46,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:47,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:48,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:35:48,618 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:49,618 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:50,618 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:51,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:35:51,061 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:35:51,062 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:35:51,063 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:35:51,618 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:51,651 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:35:52,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:35:52,293 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:35:52,618 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:52,652 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:35:53,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:35:53,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:54,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:55,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:56,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:57,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:58,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:35:59,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:35:59,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:00,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:01,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:02,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:03,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:04,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:05,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:36:05,471 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:36:05,473 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:36:05,474 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:36:05,475 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:36:05,662 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:36:05,757 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:06,663 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:36:06,753 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:07,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:36:07,293 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:36:07,898 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:08,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:09,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:10,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:11,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:36:11,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:12,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:13,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:14,676 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:36:14,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:15,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:16,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:36:16,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:17,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:18,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:19,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:36:19,743 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:36:19,744 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:36:19,746 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:36:20,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:20,674 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:36:20,675 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:36:20,925 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:21,783 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:36:21,925 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:22,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:36:22,293 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:36:22,925 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:23,925 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:24,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:25,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:26,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:27,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:36:27,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:28,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:29,971 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:30,971 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:31,971 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:32,971 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:33,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:36:33,895 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:36:33,897 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:36:33,898 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:36:33,899 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:36:34,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:34,685 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:36:34,686 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:36:35,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:36,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:37,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:36:37,293 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:36:37,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:38,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:38,463 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:36:39,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:40,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:41,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:42,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:43,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:44,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:44,398 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:36:44,680 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:36:45,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:46,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:47,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:48,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:48,470 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:36:48,472 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:36:48,473 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:36:48,474 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:36:48,695 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:36:49,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:49,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:36:50,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:50,696 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:36:51,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:52,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:52,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:36:52,331 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:36:53,316 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:54,316 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:55,316 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:55,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:36:56,316 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:57,316 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:58,316 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:36:59,316 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:00,317 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:01,317 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:01,398 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:37:02,317 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:03,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:37:03,069 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:37:03,069 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:37:03,071 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:37:03,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:03,706 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:37:04,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:04,707 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:37:05,465 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:06,463 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:07,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:37:07,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:37:07,328 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:37:07,463 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:08,464 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:09,464 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:10,464 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:11,464 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:12,398 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:37:12,464 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:13,464 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:14,464 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:14,683 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:37:15,465 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:16,465 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:17,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:37:17,425 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:37:17,426 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:37:17,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:37:17,427 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:37:17,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:17,718 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:37:18,610 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:18,718 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:37:19,610 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:20,610 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:21,611 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:22,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:37:22,328 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:37:22,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:37:22,729 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:23,718 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:24,718 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:25,718 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:26,718 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:27,719 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:28,399 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:37:28,719 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:29,719 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:30,719 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:31,719 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:32,043 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:37:32,045 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:37:32,046 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:37:32,047 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:37:32,719 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:32,734 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:37:32,735 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:37:33,719 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:34,399 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:37:34,719 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:35,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:36,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:37,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:37:37,329 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:37:37,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:38,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:39,457 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:37:39,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:40,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:41,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:42,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:43,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:44,687 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:37:44,690 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:37:44,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:45,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:46,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:37:46,523 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:37:46,524 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:37:46,525 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:37:46,744 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:37:46,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:47,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:48,746 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:37:48,920 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:49,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:50,399 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:37:50,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:51,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:52,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:37:52,328 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:37:52,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:53,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:54,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:55,482 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:37:55,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:56,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:57,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:58,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:37:59,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:00,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:01,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:38:01,139 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:38:01,139 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:38:01,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:38:01,141 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:38:01,755 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:38:01,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:02,757 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:38:02,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:03,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:04,973 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:05,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:06,400 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:38:06,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:07,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:38:07,328 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:38:07,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:08,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:09,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:10,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:11,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:12,400 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:38:12,965 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:13,965 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:14,691 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:38:14,965 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:15,681 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:38:15,682 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:38:15,683 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:38:15,688 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:38:15,767 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:38:16,144 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:16,768 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:38:17,143 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:18,179 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:18,400 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:38:19,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:20,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:21,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:22,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:22,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:38:22,328 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:38:23,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:23,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:38:24,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:25,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:26,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:27,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:28,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:29,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:29,400 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:38:30,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:30,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:38:30,511 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:38:30,512 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:38:30,513 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:38:30,778 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:38:31,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:32,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:32,780 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:38:33,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:34,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:35,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:35,400 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:38:36,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:37,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:38:37,329 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:38:37,369 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:38,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:39,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:40,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:40,462 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:38:41,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:42,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:43,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:44,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:44,695 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:38:45,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:38:45,254 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:38:45,254 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:38:45,256 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:38:45,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:45,790 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:38:46,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:38:46,527 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:46,791 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:38:47,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:48,649 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:49,649 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:50,649 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:51,401 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:38:51,649 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:52,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:38:52,329 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:38:52,649 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:53,650 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:54,650 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:55,650 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:56,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:38:56,650 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:57,650 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:58,650 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:59,650 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:38:59,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:38:59,932 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:38:59,932 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:38:59,934 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:39:00,651 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:00,801 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:39:00,802 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:39:01,651 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:01,972 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:39:02,651 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:03,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:04,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:05,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:06,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:07,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:39:07,329 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:39:07,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:39:07,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:08,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:09,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:10,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:11,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:12,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:13,401 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:39:13,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:14,698 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:39:14,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:14,833 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:39:14,834 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:39:14,835 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:39:14,836 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:39:15,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:15,813 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:39:16,750 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:16,814 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:39:17,750 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:18,750 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:19,401 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:39:19,750 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:20,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:21,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:22,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:39:22,329 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:39:22,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:23,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:24,472 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:39:24,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:25,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:26,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:27,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:28,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:29,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:39:29,674 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:39:29,675 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:39:29,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:39:29,676 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:39:29,823 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:39:29,949 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:30,824 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:39:30,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:31,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:32,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:33,861 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:34,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:35,402 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:39:35,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:36,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:37,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:39:37,329 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:39:37,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:38,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:39,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:40,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:39:40,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:41,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:42,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:43,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:44,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:39:44,658 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:39:44,658 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:39:44,660 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:39:44,702 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:39:44,835 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:39:44,932 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:45,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:39:45,931 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:46,836 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:39:47,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:48,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:49,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:50,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:51,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:51,403 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:39:52,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:52,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:39:52,329 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:39:53,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:54,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:55,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:56,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:56,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:39:57,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:58,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:59,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:39:59,855 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:39:59,857 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:39:59,857 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:39:59,859 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:40:00,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:00,847 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:40:00,848 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:40:01,133 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:01,897 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:40:02,260 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:03,254 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:04,254 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:05,254 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:06,254 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:07,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:07,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:40:07,329 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:40:07,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:40:08,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:09,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:10,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:11,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:12,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:13,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:13,403 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:40:14,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:14,705 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:40:14,778 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:40:14,779 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:40:14,779 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:40:14,781 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:40:14,858 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:40:15,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:16,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:16,860 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:40:17,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:18,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:19,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:19,403 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:40:20,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:21,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:22,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:22,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:40:22,330 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:40:23,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:24,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:24,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:40:25,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:26,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:27,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:28,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:29,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:29,823 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:40:29,825 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:40:29,826 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:40:29,826 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:40:29,827 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:40:29,869 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:40:30,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:30,870 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:40:31,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:32,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:33,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:34,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:35,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:35,403 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:40:36,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:37,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:37,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:40:37,330 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:40:38,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:39,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:40,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:40,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:40:41,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:42,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:43,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:44,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:44,709 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:40:44,855 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:40:44,856 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:40:44,857 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:40:44,858 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:40:44,880 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:40:45,260 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:45,897 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:40:46,260 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:46,882 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:40:47,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:48,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:49,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:50,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:51,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:51,404 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:40:52,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:52,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:40:52,330 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:40:53,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:54,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:55,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:56,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:56,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:40:57,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:58,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:59,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:40:59,898 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:40:59,900 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:40:59,900 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:40:59,902 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:41:00,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:00,892 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:41:00,892 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:41:01,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:01,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:41:02,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:03,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:04,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:05,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:06,309 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:07,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:07,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:41:07,330 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:41:07,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:41:08,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:09,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:10,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:11,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:12,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:13,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:13,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:41:14,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:14,713 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:41:14,978 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:41:14,980 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:41:14,980 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:41:14,982 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:41:15,309 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:15,903 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:41:16,309 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:16,904 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:41:17,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:18,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:18,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:41:19,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:20,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:21,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:22,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:41:22,330 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:41:22,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:23,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:23,470 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:41:24,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:25,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:26,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:27,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:28,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:29,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:29,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:41:30,002 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:41:30,004 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:41:30,004 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:41:30,006 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:41:30,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:30,915 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:41:30,916 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:41:31,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:32,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:33,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:34,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:34,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:41:35,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:36,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:37,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:41:37,330 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:41:37,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:38,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:39,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:39,432 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:41:40,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:41,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:42,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:43,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:44,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:44,717 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:41:44,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:41:45,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:41:45,055 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:41:45,055 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:41:45,057 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:41:45,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:45,927 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:41:46,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:46,928 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:41:47,473 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:48,472 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:49,472 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:50,472 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:50,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:41:51,472 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:52,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:41:52,331 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:41:52,473 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:53,473 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:54,473 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:55,473 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:55,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:41:56,473 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:57,473 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:58,473 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:41:59,473 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:00,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:42:00,056 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:42:00,056 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:42:00,058 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:42:00,473 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:00,938 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:42:00,938 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:42:01,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:42:01,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:02,473 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:03,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:04,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:05,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:06,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:42:06,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:07,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:42:07,331 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:42:07,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:08,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:09,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:10,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:11,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:11,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:42:12,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:13,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:14,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:14,720 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:42:15,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:42:15,287 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:42:15,288 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:42:15,289 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:42:15,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:15,951 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:42:16,559 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:16,953 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:42:17,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:42:17,707 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:18,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:19,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:20,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:21,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:22,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:42:22,331 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:42:22,466 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:42:22,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:23,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:24,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:25,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:26,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:27,473 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:42:27,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:28,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:29,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:30,431 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:42:30,433 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:42:30,433 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:42:30,435 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:42:30,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:30,963 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:42:30,964 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:42:31,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:32,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:42:32,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:33,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:34,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:35,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:36,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:37,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:42:37,331 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:42:37,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:42:37,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:38,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:39,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:40,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:41,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:42,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:43,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:42:43,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:44,724 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:42:44,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:45,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:42:45,479 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:42:45,479 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:42:45,481 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:42:45,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:45,974 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:42:46,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:46,975 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:42:47,911 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:48,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:49,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:42:49,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:50,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:51,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:52,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:42:52,331 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:42:52,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:53,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:54,490 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:42:54,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:55,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:56,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:57,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:58,911 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:42:59,911 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:00,393 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:43:00,395 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:43:00,396 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:43:00,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:43:00,397 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:43:00,911 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:00,986 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:43:00,987 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:43:01,911 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:02,911 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:03,911 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:04,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:05,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:43:05,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:06,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:07,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:43:07,331 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:43:07,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:08,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:09,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:10,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:43:10,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:11,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:12,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:13,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:14,727 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:43:14,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:15,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:43:15,695 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:43:15,696 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:43:15,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:43:15,697 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:43:15,997 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:43:16,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:16,998 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:43:17,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:18,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:19,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:20,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:21,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:21,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:43:22,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:22,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:43:22,331 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:43:23,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:24,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:25,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:26,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:27,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:27,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:43:28,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:29,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:30,179 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:30,984 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:43:30,986 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:43:30,987 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:43:30,988 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:43:31,008 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:43:31,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:32,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:32,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:43:33,009 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:43:33,390 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:34,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:35,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:36,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:37,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:43:37,331 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:43:37,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:38,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:38,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:43:39,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:40,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:41,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:42,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:43,383 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:43,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:43:44,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:44,731 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:43:45,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:46,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:43:46,241 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:43:46,242 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:43:46,243 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:43:46,513 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:47,020 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:43:47,021 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:43:47,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:48,641 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:49,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:43:49,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:50,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:51,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:52,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:43:52,332 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:43:52,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:53,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:54,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:43:54,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:55,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:56,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:57,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:58,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:43:59,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:00,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:44:00,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:01,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:44:01,463 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:44:01,463 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:44:01,465 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:44:01,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:02,031 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:44:02,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:03,032 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:44:03,871 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:04,869 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:05,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:44:05,870 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:06,870 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:07,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:44:07,332 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:44:07,870 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:08,870 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:09,870 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:10,870 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:11,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:44:11,870 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:12,871 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:13,870 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:14,736 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:44:14,870 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:15,871 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:16,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:44:16,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:44:16,740 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:44:16,741 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:44:16,742 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:44:17,005 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:17,041 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:44:18,004 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:19,043 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:44:19,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:20,130 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:21,130 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:22,130 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:22,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:44:22,332 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:44:22,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:44:23,130 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:24,130 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:25,130 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:26,130 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:27,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:27,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:44:28,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:29,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:30,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:31,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:31,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:44:31,790 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:44:31,790 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:44:31,792 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:44:32,052 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:44:32,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:33,053 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:44:33,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:33,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:44:34,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:35,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:36,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:37,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:37,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:44:37,332 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:44:38,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:39,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:39,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:44:40,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:41,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:42,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:43,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:44,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:44,739 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:44:44,746 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:44:45,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:46,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:47,093 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:44:47,095 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:44:47,095 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:44:47,097 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:44:47,372 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:48,063 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:44:48,369 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:49,064 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:44:49,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:50,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:44:50,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:51,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:52,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:44:52,352 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:44:52,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:53,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:54,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:55,490 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:44:55,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:56,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:57,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:58,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:44:59,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:00,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:01,478 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:45:01,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:02,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:45:02,339 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:45:02,339 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:45:02,341 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:45:02,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:03,074 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:45:03,074 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:45:03,526 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:04,526 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:05,526 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:06,526 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:06,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:45:07,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:45:07,345 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:45:07,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:08,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:09,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:10,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:11,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:11,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:45:12,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:13,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:14,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:14,743 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:45:15,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:16,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:16,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:45:17,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:17,758 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:45:17,760 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:45:17,761 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:45:17,762 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:45:18,084 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:45:18,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:19,085 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:45:19,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:20,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:21,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:21,800 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:45:22,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:45:22,345 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:45:22,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:23,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:24,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:25,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:26,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:27,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:45:27,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:28,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:29,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:30,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:31,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:32,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:33,038 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:45:33,040 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:45:33,041 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:45:33,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:45:33,042 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:45:33,095 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:45:33,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:34,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:35,097 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:45:35,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:36,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:37,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:45:37,345 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:45:37,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:38,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:45:38,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:39,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:40,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:41,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:42,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:43,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:45:43,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:44,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:44,746 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:45:45,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:46,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:47,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:48,235 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:45:48,237 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:45:48,238 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:45:48,239 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:45:48,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:49,111 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:45:49,111 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:45:49,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:45:49,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:50,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:51,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:52,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:45:52,345 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:45:52,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:53,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:54,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:45:54,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:55,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:56,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:57,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:58,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:45:59,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:45:59,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:00,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:01,657 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:02,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:03,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:46:03,482 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:46:03,483 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:46:03,487 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:46:03,758 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:04,121 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:46:04,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:05,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:46:05,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:06,123 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:46:06,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:07,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:46:07,345 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:46:07,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:08,717 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:09,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:10,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:46:10,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:11,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:12,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:13,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:14,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:14,750 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:46:15,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:15,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:46:16,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:17,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:18,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:18,872 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:46:18,874 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:46:18,874 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:46:18,876 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:46:19,137 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:46:19,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:20,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:20,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:46:21,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:22,140 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:46:22,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:46:22,345 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:46:22,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:23,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:24,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:25,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:26,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:46:26,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:27,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:28,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:29,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:30,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:31,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:32,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:46:32,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:33,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:34,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:46:34,396 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:46:34,396 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:46:34,398 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:46:34,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:35,150 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:46:35,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:36,151 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:46:36,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:37,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:46:37,346 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:46:37,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:46:37,801 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:38,802 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:39,802 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:40,802 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:41,802 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:42,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:46:42,802 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:43,802 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:44,755 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:46:44,802 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:45,803 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:46,803 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:47,803 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:48,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:46:48,803 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:48,953 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:46:48,955 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:46:48,955 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:46:48,957 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:46:49,161 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:46:49,804 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:50,804 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:51,803 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:52,163 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:46:52,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:46:52,345 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:46:52,803 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:53,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:46:53,803 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:54,803 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:55,804 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:56,804 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:57,804 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:58,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:46:59,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:46:59,805 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:00,805 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:01,805 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:02,805 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:03,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:47:03,286 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:47:03,287 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:47:03,288 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:47:03,805 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:04,172 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:47:04,805 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:05,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:47:05,805 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:06,174 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:47:06,805 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:07,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:47:07,346 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:47:07,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:08,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:09,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:10,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:47:10,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:11,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:12,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:13,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:14,758 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:47:14,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:15,760 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:47:15,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:16,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:17,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:47:17,496 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:47:17,496 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:47:17,498 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:47:17,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:18,184 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:47:18,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:19,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:20,185 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:47:20,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:21,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:47:21,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:22,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:47:22,346 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:47:22,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:23,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:24,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:25,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:26,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:47:26,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:27,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:28,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:29,851 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:30,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:31,558 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:47:31,559 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:47:31,559 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:47:31,561 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:47:31,851 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:32,194 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:47:32,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:47:32,851 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:33,922 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:34,196 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:47:34,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:35,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:36,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:37,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:47:37,346 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:47:37,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:47:37,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:38,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:39,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:40,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:41,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:42,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:47:42,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:43,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:44,762 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:47:44,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:45,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:47:45,773 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:47:45,773 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:47:45,775 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:47:46,129 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:46,205 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:47:47,126 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:48,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:48,206 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:47:48,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:47:49,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:50,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:51,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:52,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:52,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:47:52,346 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:47:53,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:53,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:47:54,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:55,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:56,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:57,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:58,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:59,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:47:59,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:47:59,945 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:47:59,946 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:47:59,947 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:47:59,948 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:48:00,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:00,215 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:48:01,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:02,216 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:48:02,303 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:03,296 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:04,296 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:04,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:48:05,296 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:06,296 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:07,296 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:07,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:48:07,346 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:48:08,296 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:09,296 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:10,296 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:10,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:48:11,297 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:12,297 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:13,297 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:14,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:48:14,030 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:48:14,031 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:48:14,032 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:48:14,226 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:48:14,297 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:14,765 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:48:15,297 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:15,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:48:16,228 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:48:16,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:17,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:18,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:19,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:20,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:20,812 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:48:21,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:22,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:48:22,346 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:48:22,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:23,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:24,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:25,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:26,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:26,804 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:48:27,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:28,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:48:28,176 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:48:28,177 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:48:28,178 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:48:28,238 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:48:28,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:29,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:30,240 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:48:30,548 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:31,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:32,216 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:48:32,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:33,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:34,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:35,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:36,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:37,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:48:37,346 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:48:37,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:48:37,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:38,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:39,548 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:40,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:41,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:42,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:48:42,383 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:48:42,384 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:48:42,385 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:48:42,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:42,804 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:48:43,250 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:48:43,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:44,251 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:48:44,748 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:44,769 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:48:45,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:46,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:47,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:48,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:48,805 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:48:49,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:50,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:51,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:52,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:48:52,347 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:48:52,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:53,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:53,805 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:48:54,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:55,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:56,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:48:56,439 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:48:56,440 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:48:56,442 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:48:56,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:57,261 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:48:57,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:58,262 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:48:58,824 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:48:59,805 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:48:59,817 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:00,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:01,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:02,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:03,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:04,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:49:04,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:05,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:06,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:07,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:49:07,347 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:49:07,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:08,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:09,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:49:09,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:10,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:49:10,734 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:49:10,735 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:49:10,736 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:49:10,975 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:11,273 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:49:11,897 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:12,274 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:49:12,897 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:13,898 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:14,773 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:49:14,898 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:15,775 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:49:15,978 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:16,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:17,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:18,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:19,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:20,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:49:20,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:21,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:22,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:49:22,347 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:49:22,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:23,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:24,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:24,969 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:49:24,972 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:49:24,973 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:49:24,974 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:49:25,284 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:49:25,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:26,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:49:26,285 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:49:26,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:27,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:28,960 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:29,960 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:30,960 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:31,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:49:31,960 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:32,960 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:33,960 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:34,960 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:35,960 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:36,961 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:37,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:49:37,347 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:49:37,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:49:37,961 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:38,961 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:39,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:49:39,110 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:49:39,111 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:49:39,112 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:49:39,295 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:49:39,961 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:40,961 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:41,961 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:42,298 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:49:42,895 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:49:42,961 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:43,961 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:44,776 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:49:44,962 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:45,962 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:46,962 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:47,962 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:48,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:49:48,962 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:49,962 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:50,962 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:51,962 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:52,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:49:52,348 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:49:52,962 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:53,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:49:53,429 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:49:53,430 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:49:53,431 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:49:53,963 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:54,307 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:49:54,471 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:49:54,963 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:55,963 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:56,309 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:49:56,963 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:57,963 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:58,963 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:49:59,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:49:59,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:00,963 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:01,963 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:02,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:03,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:04,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:05,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:50:05,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:06,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:07,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:50:07,347 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:50:07,719 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:50:07,721 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:50:07,722 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:50:07,723 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:50:07,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:08,318 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:50:08,967 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:10,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:10,320 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:50:10,972 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:50:11,083 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:12,083 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:13,083 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:14,083 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:14,780 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:50:15,083 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:16,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:16,782 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:50:17,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:18,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:19,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:20,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:21,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:21,969 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:50:22,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:22,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:50:22,348 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:50:22,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:50:22,518 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:50:22,519 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:50:22,520 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:50:23,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:23,330 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:50:24,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:24,331 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:50:25,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:26,212 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:26,969 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:50:27,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:28,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:29,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:30,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:31,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:32,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:32,969 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:50:33,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:34,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:35,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:36,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:37,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:37,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:50:37,347 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:50:37,902 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:50:37,904 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:50:37,904 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:50:37,906 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:50:38,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:50:38,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:38,342 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:50:39,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:40,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:40,343 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:50:41,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:42,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:43,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:44,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:50:44,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:44,784 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:50:45,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:46,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:47,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:48,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:49,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:50,113 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:50:50,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:51,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:52,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:52,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:50:52,348 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:50:53,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:53,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:50:53,510 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:50:53,511 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:50:53,512 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:50:54,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:54,356 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:50:55,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:55,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:50:56,263 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:56,358 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:50:57,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:58,264 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:50:59,264 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:00,264 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:01,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:51:01,264 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:02,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:03,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:04,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:05,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:06,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:51:06,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:07,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:07,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:51:07,348 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:51:08,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:09,043 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:51:09,044 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:51:09,045 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:51:09,046 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:51:09,291 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:09,367 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:51:10,290 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:10,368 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:51:11,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:12,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:51:12,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:13,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:14,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:14,787 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:51:15,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:16,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:17,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:17,790 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:51:18,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:19,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:20,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:21,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:22,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:51:22,348 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:51:22,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:23,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:51:23,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:24,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:24,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:51:24,535 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:51:24,535 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:51:24,537 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:51:25,380 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:51:25,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:26,382 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:51:26,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:27,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:28,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:51:28,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:29,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:30,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:31,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:32,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:33,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:51:33,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:34,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:35,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:36,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:37,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:51:37,348 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:51:37,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:38,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:39,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:51:39,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:40,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:51:40,020 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:51:40,020 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:51:40,022 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:51:40,392 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:51:40,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:41,409 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:42,394 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:51:42,409 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:43,409 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:44,409 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:44,791 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:51:44,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:51:45,409 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:46,409 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:47,409 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:48,410 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:49,410 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:50,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:51:50,410 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:51,410 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:52,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:51:52,348 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:51:52,410 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:53,410 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:54,410 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:55,410 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:55,437 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:51:55,439 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:51:55,439 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:51:55,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:51:55,441 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:51:56,405 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:51:56,406 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:51:56,411 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:57,411 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:58,411 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:51:59,411 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:00,411 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:01,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:52:01,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:02,411 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:03,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:04,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:05,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:06,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:07,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:52:07,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:52:07,348 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:52:07,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:08,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:09,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:10,413 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:11,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:52:11,183 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:52:11,183 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:52:11,185 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:52:11,417 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:52:11,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:12,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:52:12,418 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:52:12,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:13,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:14,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:14,795 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:52:15,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:16,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:17,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:17,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:52:18,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:19,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:20,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:21,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:22,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:52:22,349 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:52:22,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:23,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:52:23,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:24,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:25,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:26,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:52:26,540 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:52:26,540 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:52:26,542 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:52:26,792 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:27,430 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:52:27,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:28,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:52:28,431 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:52:28,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:29,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:30,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:31,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:32,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:33,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:34,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:52:34,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:35,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:36,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:37,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:52:37,349 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:52:37,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:38,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:39,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:52:39,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:40,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:41,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:41,975 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:52:41,977 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:52:41,978 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:52:41,979 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:52:42,442 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:52:42,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:43,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:44,444 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:52:44,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:44,798 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:52:44,804 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:52:45,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:46,750 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:47,750 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:48,750 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:49,750 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:50,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:52:50,750 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:51,750 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:52,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:52:52,349 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:52:52,750 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:53,750 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:54,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:55,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:56,117 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:52:56,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:57,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:52:57,338 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:52:57,339 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:52:57,341 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:52:57,454 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:52:57,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:58,455 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:52:58,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:52:59,759 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:00,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:01,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:02,117 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:53:02,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:03,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:04,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:05,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:06,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:07,119 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:53:07,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:53:07,349 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:53:07,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:08,753 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:09,753 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:10,753 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:11,753 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:12,753 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:12,781 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:53:12,783 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:53:12,784 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:53:12,785 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:53:12,786 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:53:13,466 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:53:13,753 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:14,467 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:53:14,753 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:14,800 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:53:15,753 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:16,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:17,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:17,802 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:53:18,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:19,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:20,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:21,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:22,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:53:22,349 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:53:22,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:23,118 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:53:23,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:24,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:25,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:26,755 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:27,755 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:28,082 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:53:28,084 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:53:28,084 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:53:28,086 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:53:28,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:53:28,478 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:53:28,757 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:29,757 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:30,480 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:53:30,757 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:31,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:32,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:33,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:53:33,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:34,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:35,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:36,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:37,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:53:37,349 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:53:37,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:38,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:53:38,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:39,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:40,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:41,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:42,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:43,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:53:43,288 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:53:43,289 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:53:43,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:53:43,290 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:53:43,489 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:53:43,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:44,491 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:53:44,803 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:53:44,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:45,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:46,944 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:47,935 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:48,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:53:48,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:49,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:50,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:51,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:52,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:53:52,350 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:53:52,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:53,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:54,326 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:53:54,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:55,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:56,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:57,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:58,435 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:53:58,437 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:53:58,438 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:53:58,440 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:53:58,501 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:53:58,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:53:59,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:53:59,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:00,503 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:54:00,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:01,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:02,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:03,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:04,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:05,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:54:05,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:06,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:07,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:54:07,350 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:54:07,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:08,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:09,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:10,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:54:10,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:11,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:12,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:13,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:14,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:54:14,087 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:54:14,087 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:54:14,089 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:54:14,515 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:54:14,806 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:54:14,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:15,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:54:15,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:16,517 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:54:16,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:17,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:18,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:19,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:20,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:21,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:54:21,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:22,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:54:22,350 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:54:22,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:23,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:24,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:25,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:26,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:27,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:54:27,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:28,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:29,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:54:29,744 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:54:29,745 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:54:29,747 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:54:29,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:30,528 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:54:30,529 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:54:30,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:32,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:32,784 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:54:33,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:34,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:35,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:36,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:37,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:37,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:54:37,350 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:54:38,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:38,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:54:39,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:40,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:41,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:42,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:43,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:43,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:54:44,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:44,810 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:54:45,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:45,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:54:45,517 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:54:45,518 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:54:45,519 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:54:45,540 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:54:46,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:46,541 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:54:47,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:48,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:49,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:49,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:54:50,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:51,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:52,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:52,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:54:52,350 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:54:53,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:54,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:55,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:55,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:54:56,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:57,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:58,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:54:59,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:00,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:01,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:01,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:55:01,307 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:55:01,308 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:55:01,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:55:01,309 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:55:01,552 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:55:02,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:02,554 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:55:03,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:04,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:05,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:06,111 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:06,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:55:07,111 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:07,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:55:07,350 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:55:08,111 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:09,111 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:10,111 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:11,111 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:12,111 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:12,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:55:13,111 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:14,111 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:14,814 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:55:15,111 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:16,111 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:17,037 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:55:17,039 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:55:17,039 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:55:17,041 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:55:17,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:17,566 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:55:18,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:55:18,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:18,567 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:55:19,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:20,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:21,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:22,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:22,350 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:55:22,350 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:55:23,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:23,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:55:24,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:25,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:26,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:27,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:28,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:28,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:55:29,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:30,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:31,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:32,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:32,954 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:55:32,956 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:55:32,956 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:55:32,958 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:55:33,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:33,576 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:55:34,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:34,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:55:34,578 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:55:35,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:36,321 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:37,321 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:37,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:55:37,350 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:55:38,321 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:39,321 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:40,321 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:40,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:55:41,321 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:42,321 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:43,321 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:44,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:44,818 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:55:45,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:45,820 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:55:46,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:47,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:48,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:48,873 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:55:48,875 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:55:48,876 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:55:48,877 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:55:49,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:49,588 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:55:50,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:50,589 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:55:50,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:55:51,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:52,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:52,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:55:52,350 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:55:53,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:54,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:55,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:56,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:56,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:55:57,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:58,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:55:59,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:00,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:01,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:01,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:56:02,324 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:03,324 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:04,324 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:04,750 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:56:04,752 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:56:04,752 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:56:04,754 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:56:05,324 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:05,600 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:56:06,324 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:06,601 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:56:06,792 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:56:07,324 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:07,350 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:56:07,350 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:56:08,324 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:09,325 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:10,325 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:11,325 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:12,325 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:12,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:56:13,325 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:14,325 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:14,821 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:56:15,325 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:16,325 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:17,325 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:17,824 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:56:18,325 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:19,326 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:20,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:56:20,280 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:56:20,280 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:56:20,282 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:56:20,534 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:20,612 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:56:21,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:22,350 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:56:22,350 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:56:22,614 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:56:22,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:23,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:56:23,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:24,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:25,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:26,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:27,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:28,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:56:28,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:29,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:30,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:31,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:32,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:33,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:34,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:56:34,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:35,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:35,866 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:56:35,867 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:56:35,868 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:56:35,870 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:56:36,625 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:56:36,626 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:56:36,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:37,350 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:56:37,351 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:56:37,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:38,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:39,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:56:39,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:40,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:41,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:42,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:43,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:44,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:44,825 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:56:44,829 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:56:45,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:46,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:47,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:48,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:49,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:50,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:56:50,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:51,372 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:56:51,374 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:56:51,374 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:56:51,375 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:56:51,637 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:56:51,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:52,350 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:56:52,351 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:56:52,638 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:56:52,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:53,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:54,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:55,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:56,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:56:56,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:57,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:58,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:56:59,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:00,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:01,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:02,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:57:02,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:03,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:04,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:05,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:06,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:06,945 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:57:06,946 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:57:06,947 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:57:06,949 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:57:07,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:57:07,351 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:57:07,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:57:07,649 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:57:07,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:08,650 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:57:08,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:09,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:10,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:11,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:12,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:13,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:57:13,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:14,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:14,829 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:57:15,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:16,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:17,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:18,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:19,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:57:19,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:20,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:21,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:22,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:57:22,352 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:57:22,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:57:22,554 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:57:22,555 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:57:22,557 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:57:22,661 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:57:22,802 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:23,799 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:24,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:57:24,662 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:57:24,921 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:25,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:26,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:27,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:28,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:29,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:30,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:57:30,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:31,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:32,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:33,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:34,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:35,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:57:35,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:36,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:37,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:57:37,351 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:57:37,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:38,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:57:38,174 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:57:38,175 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:57:38,177 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:57:38,673 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:57:38,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:39,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:40,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:57:40,675 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:57:40,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:41,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:42,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:43,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:44,834 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:57:44,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:45,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:57:45,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:46,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:47,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:48,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:49,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:50,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:51,380 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:57:51,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:52,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:57:52,352 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:57:52,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:53,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:57:53,942 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:57:53,943 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:57:53,944 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:57:53,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:54,685 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:57:54,685 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:57:54,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:55,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:56,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:56,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:57:57,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:58,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:57:59,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:00,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:01,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:02,380 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:58:02,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:03,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:04,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:05,944 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:06,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:07,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:58:07,352 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:58:07,462 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:58:07,944 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:08,944 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:09,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:58:09,864 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:58:09,865 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:58:09,867 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:58:10,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:10,696 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:58:10,697 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:58:11,035 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:12,035 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:12,905 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:58:13,036 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:14,035 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:14,836 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:58:15,093 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:16,083 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:17,083 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:18,083 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:18,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:58:19,083 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:20,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:21,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:22,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:22,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:58:22,352 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:58:23,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:23,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:58:24,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:25,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:25,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:58:25,629 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:58:25,630 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:58:25,631 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:58:25,708 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:58:26,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:26,709 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:58:27,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:28,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:28,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:58:29,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:30,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:31,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:32,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:33,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:33,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:58:34,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:35,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:36,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:37,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:37,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:58:37,352 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:58:38,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:39,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:39,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:58:40,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:41,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:41,093 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 20:58:41,095 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 20:58:41,095 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 20:58:41,097 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 20:58:41,720 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 20:58:42,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:42,721 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:58:43,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:44,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:44,840 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:58:44,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:58:45,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:46,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:47,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:48,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:49,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:50,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:50,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:58:51,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:52,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:52,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:58:52,352 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:58:53,088 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:54,088 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:55,088 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:56,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:56,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:58:57,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:58,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:58:59,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:00,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:01,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:02,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:02,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:59:03,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:04,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:05,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:06,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:07,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:07,352 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:59:07,352 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:59:07,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:59:08,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:09,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:10,740 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 20:59:12,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:13,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:59:14,843 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:59:18,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:59:19,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:20,895 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:22,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:24,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:59:29,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:59:29,789 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:30,111 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:59:30,111 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:59:32,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:34,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:59:39,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:59:40,338 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:41,608 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 20:59:41,609 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 20:59:43,113 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:44,847 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 20:59:44,853 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:59:49,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:59:51,367 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:53,221 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 20:59:54,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 20:59:59,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:00:00,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:01,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:04,830 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:05,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:00:10,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:00:10,920 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:00:10,920 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:00:14,851 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:00:15,853 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:00:16,342 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:00:16,343 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:00:16,383 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:16,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:17,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:18,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:18,980 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:00:19,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:20,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:20,928 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:00:21,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:22,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:23,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:24,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:25,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:26,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:26,490 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:00:27,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:28,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:29,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:30,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:31,341 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:00:31,341 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:00:31,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:31,507 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:00:32,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:33,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:34,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:35,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:36,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:37,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:00:37,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:38,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:39,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:40,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:41,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:42,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:43,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:00:43,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:44,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:44,855 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:00:45,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:46,341 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:00:46,342 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:00:46,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:47,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:48,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:00:48,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:49,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:50,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:51,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:52,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:53,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:54,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:00:54,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:55,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:56,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:57,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:58,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:00:59,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:00:59,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:00,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:01,341 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:01:01,342 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:01:01,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:02,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:03,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:04,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:05,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:01:05,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:06,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:07,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:08,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:09,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:10,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:11,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:01:11,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:12,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:13,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:14,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:14,858 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:01:15,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:16,341 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:01:16,342 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:01:16,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:17,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:01:17,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:22,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:01:27,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:01:32,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:01:35,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:35,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:01:35,739 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:01:36,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:37,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:37,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:01:38,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:39,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:40,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:41,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:42,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:43,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:43,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:01:44,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:44,861 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:01:45,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:46,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:47,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:48,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:49,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:01:49,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:50,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:50,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:01:50,737 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:01:51,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:52,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:53,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:54,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:54,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:01:55,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:56,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:57,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:58,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:01:59,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:00,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:02:00,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:01,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:02,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:03,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:04,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:05,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:05,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:02:05,737 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:02:05,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:02:06,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:07,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:08,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:09,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:10,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:11,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:02:11,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:12,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:13,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:14,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:14,865 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:02:15,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:16,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:17,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:02:17,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:18,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:19,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:20,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:02:20,737 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:02:20,778 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:21,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:22,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:22,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:02:23,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:24,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:25,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:26,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:27,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:28,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:02:28,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:29,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:30,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:31,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:32,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:33,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:34,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:02:34,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:35,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:02:35,738 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:02:35,778 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:36,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:37,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:38,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:39,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:39,898 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:02:40,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:41,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:42,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:43,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:44,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:44,869 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:02:45,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:02:45,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:46,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:47,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:48,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:49,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:50,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:02:50,738 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:02:50,778 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:50,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:02:51,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:52,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:53,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:54,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:55,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:56,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:02:56,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:57,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:58,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:02:59,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:00,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:01,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:03:01,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:02,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:03,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:04,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:05,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:03:05,738 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:03:05,778 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:06,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:06,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:03:07,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:08,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:09,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:10,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:11,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:12,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:03:12,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:13,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:14,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:14,872 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:03:15,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:16,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:17,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:03:17,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:18,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:19,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:20,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:03:20,738 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:03:20,778 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:21,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:22,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:22,830 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:03:23,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:24,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:25,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:26,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:27,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:28,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:03:28,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:29,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:30,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:31,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:32,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:03:34,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:03:39,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:03:44,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:03:44,876 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:03:49,878 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:03:54,879 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:03:59,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:04:04,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:04:09,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:04:11,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:04:11,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:11,648 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:04:12,641 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:13,642 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:14,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:14,881 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:04:14,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:04:15,251 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:04:15,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:16,822 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:17,253 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:04:17,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:18,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:19,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:20,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:04:20,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:21,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:22,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:23,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:24,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:25,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:04:25,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:26,645 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:04:26,646 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:04:26,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:27,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:28,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:29,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:30,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:04:30,631 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:04:30,631 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:04:30,632 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:04:30,671 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:04:30,871 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:31,263 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:04:31,870 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:32,961 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:33,265 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:04:33,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:34,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:35,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:36,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:04:36,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:37,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:38,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:39,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:40,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:41,645 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:04:41,646 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:04:41,780 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:04:41,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:42,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:43,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:44,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:04:44,200 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:04:44,200 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:04:44,202 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:04:44,273 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:04:44,882 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:04:44,962 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:45,274 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:04:45,961 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:46,961 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:47,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:04:47,995 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:48,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:49,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:50,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:51,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:52,648 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:04:52,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:53,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:54,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:55,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:56,645 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:04:56,646 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:04:56,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:57,749 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:04:57,929 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:04:57,931 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:04:57,931 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:04:57,933 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:04:58,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:58,284 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:04:59,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:04:59,285 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:05:00,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:01,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:02,081 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:03,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:03,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:05:04,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:05,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:06,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:07,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:08,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:09,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:09,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:05:10,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:11,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:11,555 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:05:11,557 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:05:11,558 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:05:11,559 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:05:11,795 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:05:11,796 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:05:12,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:12,294 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:05:13,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:13,296 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:05:14,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:14,885 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:05:14,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:05:15,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:16,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:17,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:18,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:19,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:20,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:20,796 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:05:21,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:22,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:23,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:24,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:25,159 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:05:25,160 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:05:25,161 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:05:25,163 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:05:25,305 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:05:25,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:26,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:05:26,317 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:26,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:05:26,710 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:05:27,307 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:05:27,317 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:28,318 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:29,318 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:30,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:31,368 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:31,796 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:05:32,369 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:33,369 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:34,369 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:35,369 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:36,369 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:36,796 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:05:37,369 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:38,369 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:38,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:05:38,860 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:05:38,861 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:05:38,863 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:05:39,316 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:05:39,369 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:40,370 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:41,317 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:05:41,370 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:41,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:05:41,710 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:05:41,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:05:42,370 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:43,370 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:44,370 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:44,889 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:05:45,370 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:46,370 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:47,370 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:47,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:05:48,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:49,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:50,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:51,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:52,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:52,578 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:05:52,579 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:05:52,580 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:05:52,582 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:05:53,327 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:05:53,328 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:05:53,373 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:53,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:05:54,373 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:55,373 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:56,373 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:56,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:05:56,710 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:05:57,373 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:58,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:05:58,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:05:59,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:00,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:01,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:02,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:03,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:04,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:04,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:06:05,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:06,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:06,430 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:06:06,432 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:06:06,432 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:06:06,434 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:06:07,340 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:06:07,341 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:06:07,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:08,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:09,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:10,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:10,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:06:11,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:11,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:06:11,710 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:06:12,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:13,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:14,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:14,893 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:06:15,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:15,895 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:06:16,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:17,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:18,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:19,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:20,043 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:06:20,044 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:06:20,045 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:06:20,046 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:06:20,351 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:06:20,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:21,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:06:21,352 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:06:21,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:22,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:23,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:24,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:25,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:26,398 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:26,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:06:26,710 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:06:26,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:06:27,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:28,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:29,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:30,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:31,398 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:32,398 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:32,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:06:33,398 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:33,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:06:33,629 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:06:33,630 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:06:33,631 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:06:34,362 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:06:34,398 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:35,363 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:06:35,398 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:36,398 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:37,398 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:38,398 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:38,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:06:39,398 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:40,398 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:41,399 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:41,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:06:41,711 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:06:42,399 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:43,399 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:43,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:06:44,399 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:44,897 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:06:45,399 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:46,399 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:47,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:06:47,155 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:06:47,156 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:06:47,158 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:06:47,373 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:06:47,399 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:48,399 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:49,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:06:49,375 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:06:49,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:50,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:51,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:52,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:53,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:54,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:54,799 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:06:55,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:56,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:56,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:06:56,711 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:06:57,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:58,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:59,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:06:59,873 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:07:00,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:00,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:07:00,887 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:07:00,888 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:07:00,889 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:07:01,384 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:07:01,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:02,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:03,385 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:07:03,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:04,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:05,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:05,802 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:07:06,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:07,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:08,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:09,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:10,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:11,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:11,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:07:11,711 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:07:11,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:07:12,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:13,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:14,570 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:07:14,571 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:07:14,572 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:07:14,573 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:07:14,805 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:14,900 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:07:15,395 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:07:15,396 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:07:15,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:16,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:16,902 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:07:17,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:18,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:19,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:20,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:21,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:21,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:07:22,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:23,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:24,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:25,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:26,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:07:26,711 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:07:26,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:26,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:07:27,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:28,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:07:28,158 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:07:28,159 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:07:28,160 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:07:28,405 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:07:28,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:29,406 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:07:29,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:30,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:31,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:32,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:32,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:07:33,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:34,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:35,726 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:36,726 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:37,726 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:37,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:07:38,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:39,726 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:40,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:41,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:07:41,711 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:07:41,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:41,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:07:41,780 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:07:41,781 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:07:41,782 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:07:42,416 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:07:42,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:42,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:07:43,417 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:07:43,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:44,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:44,903 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:07:45,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:46,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:47,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:48,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:48,984 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:07:49,728 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:50,728 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:51,728 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:52,728 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:53,728 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:53,984 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:07:54,728 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:55,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:07:55,498 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:07:55,498 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:07:55,500 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:07:55,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:56,427 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:07:56,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:07:56,711 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:07:56,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:57,428 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:07:57,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:58,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:59,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:07:59,984 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:08:00,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:01,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:02,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:03,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:04,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:05,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:05,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:08:06,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:07,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:08,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:09,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:08:09,179 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:08:09,179 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:08:09,181 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:08:09,437 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:08:09,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:10,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:11,219 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:08:11,439 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:08:11,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:08:11,711 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:08:11,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:12,851 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:13,844 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:14,844 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:14,907 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:08:15,844 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:16,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:16,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:08:17,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:18,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:19,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:20,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:21,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:21,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:08:22,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:08:22,690 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:08:22,691 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:08:22,692 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:08:22,932 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:23,449 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:08:23,449 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:08:23,931 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:25,035 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:26,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:26,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:08:26,711 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:08:27,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:27,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:08:28,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:29,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:30,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:31,035 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:32,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:32,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:08:33,035 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:34,035 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:35,035 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:36,035 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:36,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:08:36,314 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:08:36,314 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:08:36,316 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:08:36,459 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:08:37,035 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:37,460 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:08:38,035 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:38,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:08:39,035 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:40,060 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:41,058 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:41,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:08:41,711 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:08:42,058 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:43,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:44,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:44,911 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:08:44,918 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:08:45,060 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:46,060 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:47,060 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:48,060 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:49,060 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:49,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:08:50,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:08:50,055 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:08:50,055 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:08:50,057 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:08:50,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:50,469 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:08:51,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:51,470 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:08:52,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:53,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:54,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:55,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:55,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:08:56,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:56,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:08:56,711 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:08:57,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:58,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:08:59,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:00,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:01,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:01,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:09:02,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:03,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:03,570 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:09:03,572 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:09:03,572 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:09:03,574 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:09:04,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:04,479 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:09:05,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:05,480 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:09:06,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:06,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:09:07,304 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:08,297 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:09,297 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:10,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:11,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:11,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:09:11,712 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:09:12,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:12,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:09:13,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:14,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:14,915 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:09:15,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:16,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:17,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:17,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:09:17,329 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:09:17,330 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:09:17,332 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:09:17,490 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:09:18,300 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:18,368 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:09:19,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:19,492 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:09:20,300 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:21,300 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:22,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:23,447 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:23,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:09:24,447 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:25,447 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:26,447 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:26,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:09:26,712 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:09:27,447 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:28,447 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:29,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:29,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:09:30,447 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:30,965 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:09:30,967 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:09:30,967 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:09:30,968 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:09:31,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:31,501 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:09:31,502 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:09:32,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:33,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:34,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:35,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:35,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:09:36,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:37,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:38,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:39,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:40,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:40,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:09:41,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:41,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:09:41,712 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:09:42,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:43,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:44,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:44,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:09:44,708 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:09:44,708 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:09:44,710 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:09:44,918 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:09:45,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:45,512 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:09:45,512 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:09:46,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:46,920 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:09:47,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:48,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:49,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:50,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:51,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:52,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:09:52,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:53,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:54,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:55,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:56,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:56,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:09:56,712 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:09:57,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:58,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:09:58,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:09:58,414 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:09:58,414 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:09:58,416 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:09:58,522 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:09:58,641 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:09:59,523 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:09:59,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:00,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:01,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:02,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:03,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:10:03,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:04,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:05,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:06,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:07,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:08,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:09,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:10:09,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:10,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:11,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:11,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:10:11,712 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:10:12,050 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:10:12,052 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:10:12,053 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:10:12,055 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:10:12,532 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:10:12,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:13,533 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:10:13,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:14,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:10:14,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:14,922 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:10:15,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:16,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:17,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:18,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:19,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:20,234 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:10:20,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:21,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:22,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:23,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:24,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:25,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:25,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:10:25,770 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:10:25,771 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:10:25,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:10:25,773 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:10:26,543 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:10:26,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:26,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:10:26,712 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:10:27,544 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:10:27,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:28,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:29,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:30,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:31,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:10:31,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:32,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:33,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:34,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:35,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:36,569 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:37,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:10:37,569 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:38,569 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:39,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:10:39,417 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:10:39,417 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:10:39,419 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:10:39,553 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:10:39,652 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:40,649 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:41,555 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:10:41,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:41,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:10:41,764 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:10:42,755 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:43,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:10:43,755 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:44,755 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:44,925 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:10:45,755 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:46,755 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:47,755 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:48,232 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:10:48,755 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:49,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:50,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:51,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:52,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:53,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:10:53,182 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:10:53,183 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:10:53,184 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:10:53,563 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:10:53,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:54,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:10:54,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:55,565 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:10:55,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:56,907 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:10:56,908 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:10:56,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:57,906 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:58,906 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:10:59,232 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:10:59,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:00,911 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:01,911 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:02,911 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:03,911 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:04,232 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:11:04,911 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:05,911 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:06,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:11:06,850 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:11:06,850 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:11:06,852 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:11:07,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:07,574 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:11:07,574 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:11:07,994 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:08,995 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:09,995 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:10,232 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:11:10,995 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:12,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:12,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:11:12,017 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:11:13,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:14,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:14,929 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:11:15,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:15,931 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:11:16,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:17,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:18,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:19,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:20,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:20,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:11:20,530 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:11:20,530 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:11:20,532 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:11:20,583 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:11:21,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:21,570 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:11:21,584 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:11:22,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:23,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:24,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:25,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:26,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:27,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:11:27,014 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:11:27,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:27,144 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:11:28,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:29,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:30,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:31,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:32,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:32,233 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:11:33,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:34,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:34,220 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:11:34,222 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:11:34,222 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:11:34,224 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:11:34,595 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:11:35,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:36,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:36,597 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:11:37,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:37,233 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:11:38,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:39,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:40,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:41,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:42,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:11:42,014 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:11:42,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:43,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:43,233 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:11:44,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:44,932 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:11:45,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:46,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:47,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:47,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:11:47,766 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:11:47,767 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:11:47,768 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:11:48,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:48,605 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:11:48,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:11:49,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:50,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:50,606 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:11:51,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:52,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:53,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:54,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:54,233 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:11:55,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:56,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:57,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:11:57,014 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:11:57,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:58,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:59,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:11:59,234 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:12:00,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:01,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:01,339 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:12:01,341 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:12:01,341 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:12:01,343 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:12:01,615 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:12:02,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:03,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:04,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:04,617 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:12:05,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:05,234 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:12:06,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:07,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:08,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:09,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:10,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:11,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:11,234 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:12:12,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:12:12,015 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:12:12,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:13,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:14,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:14,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:12:14,893 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:12:14,893 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:12:14,895 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:12:14,935 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:12:15,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:15,625 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:12:16,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:16,626 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:12:16,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:12:17,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:18,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:19,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:20,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:21,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:22,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:12:22,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:23,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:24,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:25,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:26,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:27,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:12:27,015 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:12:27,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:12:27,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:28,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:28,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:12:28,534 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:12:28,535 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:12:28,536 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:12:28,635 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:12:29,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:30,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:30,636 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:12:31,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:32,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:32,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:12:33,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:34,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:35,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:36,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:37,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:38,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:38,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:12:39,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:40,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:41,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:42,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:12:42,015 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:12:42,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:12:42,124 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:12:42,125 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:12:42,126 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:12:42,354 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:42,644 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:12:43,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:43,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:12:44,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:44,646 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:12:44,939 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:12:45,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:46,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:47,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:48,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:49,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:49,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:12:50,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:51,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:52,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:53,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:54,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:54,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:12:55,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:55,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:12:55,698 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:12:55,698 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:12:55,699 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:12:56,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:56,656 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:12:57,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:12:57,015 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:12:57,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:58,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:12:58,658 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:12:59,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:00,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:13:00,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:01,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:02,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:03,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:04,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:05,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:13:05,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:06,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:07,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:08,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:09,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:09,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:13:09,426 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:13:09,427 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:13:09,430 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:13:09,666 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:13:10,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:10,466 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:13:11,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:12,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:13:12,015 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:13:12,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:12,668 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:13:13,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:14,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:14,943 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:13:15,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:15,945 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:13:16,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:17,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:18,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:19,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:20,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:21,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:13:21,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:22,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:23,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:13:23,155 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:13:23,156 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:13:23,157 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:13:23,389 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:23,676 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:13:24,303 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:24,677 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:13:25,304 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:26,304 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:27,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:13:27,015 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:13:27,129 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:13:27,304 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:28,304 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:29,304 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:30,304 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:31,304 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:32,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:13:32,304 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:33,304 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:34,304 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:35,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:36,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:36,824 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:13:36,825 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:13:36,826 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:13:36,827 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:13:37,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:37,686 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:13:38,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:13:38,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:38,687 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:13:39,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:40,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:41,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:42,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:13:42,015 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:13:42,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:43,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:44,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:13:44,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:44,947 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:13:45,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:46,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:47,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:48,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:49,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:13:49,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:50,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:50,455 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:13:50,457 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:13:50,457 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:13:50,459 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:13:50,695 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:13:51,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:52,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:52,697 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:13:53,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:54,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:55,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:13:55,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:56,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:57,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:13:57,015 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:13:57,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:58,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:13:59,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:00,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:01,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:14:01,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:02,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:03,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:04,162 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:14:04,164 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:14:04,165 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:14:04,166 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:14:04,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:04,706 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:14:05,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:06,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:06,709 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:14:07,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:14:07,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:08,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:09,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:10,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:11,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:12,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:14:12,016 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:14:12,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:14:12,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:13,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:14,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:14,951 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:14:15,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:16,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:17,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:14:17,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:17,757 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:14:17,759 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:14:17,759 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:14:17,761 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:14:18,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:18,718 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:14:19,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:20,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:20,720 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:14:21,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:22,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:23,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:14:23,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:24,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:25,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:26,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:27,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:14:27,015 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:14:27,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:28,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:14:28,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:29,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:30,478 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:31,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:14:31,384 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:14:31,385 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:14:31,386 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:14:31,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:31,728 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:14:32,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:33,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:34,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:14:34,729 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:34,732 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:14:35,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:36,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:37,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:38,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:39,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:14:39,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:40,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:41,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:42,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:14:42,016 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:14:42,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:43,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:44,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:44,955 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:14:44,961 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:14:45,082 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:14:45,084 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:14:45,084 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:14:45,086 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:14:45,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:45,740 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:14:46,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:46,741 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:14:47,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:48,796 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:49,796 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:50,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:14:50,795 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:51,795 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:52,795 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:53,795 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:54,796 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:55,796 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:56,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:14:56,796 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:57,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:14:57,016 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:14:57,796 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:58,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:14:58,734 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:14:58,734 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:14:58,736 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:14:58,750 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:14:58,966 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:14:59,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:00,751 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:15:00,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:01,774 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:15:01,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:02,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:03,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:04,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:05,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:06,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:07,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:15:07,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:08,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:09,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:10,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:11,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:12,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:15:12,016 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:15:12,431 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:15:12,433 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:15:12,433 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:15:12,435 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:15:12,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:15:12,761 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:15:12,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:13,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:14,762 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:15:14,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:14,958 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:15:15,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:16,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:17,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:18,592 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:15:18,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:19,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:20,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:21,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:22,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:23,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:24,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:15:24,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:25,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:26,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:15:26,068 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:15:26,069 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:15:26,070 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:15:26,772 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:15:26,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:27,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:15:27,016 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:15:27,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:28,774 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:15:28,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:29,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:15:29,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:30,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:31,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:32,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:33,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:34,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:35,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:15:35,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:36,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:37,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:38,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:39,697 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:15:39,699 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:15:39,700 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:15:39,701 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:15:39,782 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:15:39,932 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:40,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:15:40,783 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:15:40,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:42,032 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:15:42,033 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:15:42,073 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:43,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:44,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:44,961 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:15:45,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:45,963 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:15:46,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:47,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:48,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:49,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:50,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:51,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:51,590 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:15:52,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:53,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:53,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:15:53,253 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:15:53,254 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:15:53,255 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:15:53,792 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:15:54,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:54,793 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:15:55,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:56,032 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:57,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:15:57,026 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:15:57,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:57,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:15:58,032 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:15:59,032 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:00,032 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:01,032 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:02,033 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:02,590 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:16:03,033 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:04,033 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:05,033 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:06,033 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:06,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:16:06,732 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:16:06,733 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:16:06,735 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:16:06,802 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:16:07,033 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:08,033 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:08,590 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:16:08,804 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:16:09,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:10,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:11,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:12,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:16:12,026 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:16:12,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:13,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:14,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:14,590 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:16:14,965 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:16:15,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:16,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:17,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:18,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:19,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:20,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:20,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:16:20,155 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:16:20,155 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:16:20,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:16:20,157 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:16:20,813 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:16:21,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:22,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:22,815 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:16:23,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:24,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:25,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:25,590 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:16:26,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:27,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:16:27,026 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:16:27,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:28,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:29,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:30,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:30,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:16:31,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:32,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:33,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:33,774 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:16:33,776 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:16:33,777 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:16:33,778 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:16:33,824 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:16:34,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:34,825 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:16:35,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:35,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:16:36,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:37,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:38,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:39,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:40,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:41,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:41,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:16:42,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:16:42,027 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:16:42,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:43,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:44,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:44,969 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:16:45,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:46,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:46,971 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:16:47,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:47,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:16:47,454 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:16:47,454 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:16:47,456 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:16:47,835 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:16:48,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:48,836 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:16:49,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:50,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:51,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:52,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:52,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:16:53,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:54,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:55,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:56,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:57,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:16:57,027 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:16:57,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:57,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:16:58,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:16:59,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:00,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:01,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:17:01,159 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:17:01,160 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:17:01,161 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:17:01,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:01,845 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:17:02,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:02,845 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:17:03,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:03,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:17:04,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:05,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:06,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:07,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:08,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:09,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:09,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:17:10,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:11,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:12,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:17:12,027 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:17:12,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:13,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:14,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:14,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:17:14,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:17:14,816 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:17:14,816 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:17:14,818 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:17:14,855 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:17:14,973 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:17:15,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:16,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:16,857 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:17:17,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:18,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:19,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:19,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:17:20,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:21,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:22,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:23,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:24,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:25,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:25,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:17:26,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:27,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:17:27,027 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:17:27,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:28,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:28,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:17:28,542 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:17:28,543 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:17:28,544 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:17:28,866 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:17:29,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:30,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:30,867 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:17:31,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:31,584 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:17:32,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:33,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:34,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:35,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:36,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:36,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:17:37,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:38,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:39,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:40,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:41,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:41,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:17:42,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:17:42,027 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:17:42,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:17:42,120 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:17:42,121 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:17:42,122 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:17:42,342 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:42,876 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:17:42,877 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:17:43,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:44,434 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:44,976 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:17:45,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:46,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:46,978 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:17:47,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:48,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:49,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:50,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:51,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:52,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:52,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:17:53,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:54,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:55,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:55,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:17:55,703 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:17:55,703 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:17:55,705 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:17:55,886 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:17:56,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:56,887 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:17:57,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:17:57,027 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:17:57,430 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:57,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:17:58,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:17:59,471 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:00,471 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:01,471 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:02,471 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:03,471 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:03,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:18:04,472 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:05,472 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:06,472 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:07,472 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:08,472 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:08,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:18:09,434 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:18:09,436 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:18:09,437 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:18:09,438 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:18:09,671 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:09,897 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:18:10,587 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:10,898 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:18:11,587 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:12,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:18:12,027 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:18:12,587 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:13,587 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:14,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:14,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:18:14,980 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:18:15,587 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:16,587 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:17,587 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:18,588 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:19,588 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:19,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:18:20,588 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:21,588 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:22,588 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:23,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:18:23,091 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:18:23,092 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:18:23,093 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:18:23,588 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:23,907 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:18:24,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:24,909 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:18:25,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:18:25,588 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:26,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:27,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:18:27,028 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:18:27,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:28,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:29,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:30,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:30,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:18:31,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:32,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:33,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:34,590 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:35,590 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:35,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:18:36,590 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:36,867 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:18:36,869 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:18:36,869 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:18:36,871 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:18:36,917 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:18:37,590 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:38,590 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:38,919 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:18:39,590 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:40,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:41,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:41,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:18:42,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:18:42,029 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:18:42,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:43,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:44,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:44,983 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:18:45,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:46,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:46,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:18:47,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:48,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:49,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:50,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:50,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:18:50,691 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:18:50,692 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:18:50,693 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:18:50,928 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:18:51,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:52,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:52,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:18:52,930 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:18:53,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:54,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:55,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:56,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:57,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:18:57,029 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:18:57,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:57,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:18:58,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:18:59,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:00,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:01,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:02,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:02,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:19:03,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:04,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:04,974 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:19:04,976 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:19:04,976 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:19:04,978 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:19:05,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:05,940 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:19:06,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:06,941 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:19:07,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:08,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:19:08,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:09,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:10,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:11,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:12,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:19:12,028 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:19:12,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:13,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:13,690 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:19:14,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:14,987 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:19:15,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:16,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:17,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:18,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:18,825 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:19:18,826 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:19:18,827 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:19:18,827 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:19:18,828 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:19:18,951 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:19:19,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:20,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:20,952 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:19:21,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:22,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:23,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:24,690 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:24,691 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:19:25,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:26,690 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:27,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:19:27,029 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:19:27,690 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:28,690 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:29,690 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:29,691 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:19:30,690 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:31,690 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:32,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:19:32,430 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:19:32,430 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:19:32,432 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:19:32,690 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:32,961 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:19:33,690 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:34,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:34,963 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:19:35,690 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:19:35,759 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:36,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:37,760 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:38,760 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:39,760 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:40,691 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:19:40,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:41,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:42,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:19:42,029 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:19:42,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:43,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:44,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:44,991 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:19:45,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:45,762 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:19:45,764 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:19:45,764 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:19:45,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:19:45,766 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:19:45,971 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:19:46,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:46,972 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:19:47,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:48,762 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:49,762 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:50,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:51,691 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:19:51,855 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:52,855 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:53,855 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:54,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:55,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:56,692 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:19:56,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:57,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:19:57,029 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:19:57,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:58,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:59,060 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:19:59,062 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:19:59,062 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:19:59,064 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:19:59,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:19:59,982 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:20:00,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:00,984 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:20:01,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:02,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:20:02,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:03,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:04,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:05,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:06,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:07,762 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:20:07,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:08,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:09,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:10,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:11,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:12,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:20:12,029 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:20:12,392 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:20:12,394 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:20:12,395 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:20:12,396 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:20:12,762 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:20:12,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:12,993 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:20:13,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:14,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:14,995 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:20:15,002 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:20:15,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:16,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:17,762 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:20:17,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:18,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:19,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:20,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:21,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:22,762 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:20:22,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:23,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:24,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:25,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:20:25,709 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:20:25,710 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:20:25,711 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:20:25,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:26,011 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:20:26,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:27,012 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:20:27,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:20:27,029 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:20:28,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:28,762 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:20:29,076 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:30,076 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:31,076 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:32,076 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:33,076 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:34,076 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:34,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:20:35,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:36,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:37,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:38,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:38,978 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:20:38,980 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:20:38,980 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:20:38,981 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:20:39,021 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:20:39,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:40,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:20:40,120 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:41,023 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:20:41,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:42,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:20:42,029 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:20:42,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:43,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:44,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:44,998 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:20:45,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:46,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:20:46,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:47,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:48,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:49,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:50,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:51,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:51,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:20:52,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:52,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:20:52,229 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:20:52,229 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:20:52,231 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:20:53,032 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:20:53,033 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:20:53,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:54,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:55,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:56,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:57,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:20:57,030 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:20:57,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:57,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:20:58,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:20:59,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:00,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:01,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:02,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:02,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:21:03,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:04,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:05,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:05,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:21:05,505 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:21:05,505 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:21:05,508 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:21:06,045 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:21:06,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:07,046 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:21:07,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:07,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:21:08,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:09,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:10,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:11,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:12,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:21:12,029 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:21:12,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:13,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:13,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:21:14,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:15,002 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:21:15,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:16,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:17,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:18,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:18,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:21:18,752 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:21:18,753 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:21:18,755 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:21:19,055 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:21:19,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:19,793 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:21:20,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:21,056 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:21:21,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:22,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:23,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:24,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:24,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:21:25,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:26,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:27,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:21:27,030 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:21:27,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:28,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:29,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:30,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:30,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:21:31,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:32,095 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:21:32,097 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:21:32,097 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:21:32,099 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:21:32,303 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:33,065 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:21:33,066 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:21:33,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:34,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:35,413 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:36,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:36,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:21:37,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:38,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:39,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:40,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:41,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:42,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:21:42,030 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:21:42,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:21:42,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:43,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:44,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:45,006 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:21:45,386 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:21:45,387 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:21:45,388 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:21:45,389 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:21:45,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:46,074 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:21:46,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:47,075 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:21:47,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:21:47,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:48,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:49,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:50,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:51,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:52,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:52,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:21:53,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:54,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:55,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:56,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:57,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:21:57,030 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:21:57,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:57,809 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:21:58,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:21:58,640 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:21:58,641 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:21:58,642 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:21:58,643 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:21:59,084 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:21:59,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:00,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:01,086 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:22:01,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:02,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:03,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:03,809 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:22:04,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:05,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:06,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:07,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:08,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:08,809 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:22:09,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:10,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:11,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:12,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:22:12,030 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:22:12,070 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:22:12,122 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:22:12,123 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:22:12,125 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:22:12,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:13,094 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:22:13,095 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:22:13,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:13,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:22:14,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:15,009 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:22:15,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:16,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:17,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:18,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:19,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:19,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:22:20,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:21,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:22,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:23,525 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:24,525 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:24,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:22:25,457 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:22:25,459 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:22:25,459 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:22:25,460 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:22:25,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:26,108 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:22:26,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:27,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:22:27,030 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:22:27,109 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:22:27,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:28,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:29,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:29,868 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:22:30,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:31,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:32,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:33,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:34,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:35,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:35,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:22:36,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:37,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:38,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:38,946 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:22:38,948 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:22:38,948 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:22:38,950 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:22:39,120 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:22:39,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:40,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:40,988 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:22:41,121 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:22:41,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:42,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:22:42,030 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:22:42,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:43,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:44,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:45,013 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:22:45,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:46,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:22:46,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:47,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:48,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:49,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:50,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:51,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:51,817 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:22:52,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:22:52,314 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:22:52,314 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:22:52,316 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:22:52,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:53,130 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:22:53,131 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:22:53,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:54,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:55,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:56,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:56,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:22:57,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:22:57,031 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:22:57,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:58,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:22:59,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:00,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:01,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:02,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:02,817 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:23:03,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:04,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:05,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:05,728 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:23:05,729 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:23:05,730 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:23:05,732 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:23:06,139 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:23:06,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:07,140 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:23:07,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:08,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:08,817 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:23:09,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:10,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:11,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:12,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:23:12,031 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:23:12,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:13,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:13,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:23:14,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:15,016 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:23:15,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:16,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:17,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:18,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:19,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:23:19,118 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:23:19,118 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:23:19,119 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:23:19,120 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:23:19,148 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:23:19,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:20,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:21,150 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:23:21,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:22,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:23,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:24,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:24,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:23:25,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:26,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:27,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:23:27,031 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:23:27,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:28,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:29,670 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:30,670 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:30,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:23:31,670 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:32,580 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:23:32,581 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:23:32,582 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:23:32,583 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:23:32,794 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:33,158 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:23:33,793 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:34,905 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:35,159 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:23:35,899 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:36,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:23:36,899 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:37,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:38,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:39,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:40,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:41,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:23:41,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:42,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:23:42,031 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:23:42,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:43,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:44,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:45,020 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:23:45,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:23:45,746 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:23:45,747 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:23:45,748 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:23:46,129 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:46,168 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:23:47,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:47,169 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:23:47,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:23:48,148 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:49,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:50,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:51,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:52,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:52,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:23:53,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:54,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:55,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:56,142 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:57,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:23:57,031 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:23:57,142 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:57,833 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:23:58,142 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:23:59,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:23:59,090 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:23:59,091 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:23:59,093 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:23:59,178 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:23:59,290 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:00,221 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:01,180 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:24:01,221 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:02,221 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:03,221 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:03,833 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:24:04,222 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:05,221 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:06,222 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:07,222 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:08,222 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:09,222 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:09,833 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:24:10,222 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:11,222 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:12,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:24:12,031 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:24:12,222 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:12,392 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:24:12,394 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:24:12,394 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:24:12,396 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:24:13,190 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:24:13,190 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:24:13,222 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:14,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:15,023 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:24:15,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:24:15,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:16,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:17,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:18,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:19,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:20,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:20,833 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:24:21,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:22,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:23,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:24,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:25,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:25,584 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:24:25,585 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:24:25,586 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:24:25,588 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:24:26,200 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:24:26,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:26,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:24:27,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:24:27,031 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:24:27,201 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:24:27,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:28,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:29,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:30,326 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:31,326 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:31,946 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:24:32,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:33,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:34,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:35,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:36,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:37,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:37,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:24:38,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:38,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:24:38,983 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:24:38,984 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:24:38,988 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:24:39,210 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:24:39,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:40,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:41,212 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:24:41,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:42,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:24:42,032 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:24:42,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:43,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:43,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:24:44,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:45,027 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:24:45,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:46,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:47,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:48,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:48,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:24:49,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:50,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:51,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:52,290 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:24:52,291 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:24:52,292 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:24:52,293 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:24:52,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:53,222 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:24:53,223 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:24:53,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:53,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:24:54,611 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:55,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:56,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:57,032 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:24:57,033 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:24:57,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:58,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:59,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:24:59,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:25:00,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:01,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:02,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:03,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:04,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:04,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:25:05,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:05,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:25:05,770 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:25:05,771 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:25:05,772 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:25:06,233 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:25:06,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:07,235 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:25:07,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:08,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:09,603 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:10,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:25:10,603 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:11,603 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:12,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:25:12,032 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:25:12,603 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:13,603 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:14,603 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:15,032 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:25:15,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:16,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:25:16,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:17,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:18,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:19,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:25:19,157 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:25:19,157 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:25:19,158 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:25:19,244 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:25:19,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:20,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:21,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:25:21,246 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:25:21,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:22,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:23,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:24,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:25,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:26,605 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:27,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:25:27,032 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:25:27,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:25:27,605 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:28,605 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:29,605 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:30,605 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:31,605 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:32,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:25:32,202 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:25:32,202 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:25:32,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:25:32,204 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:25:32,254 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:25:32,605 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:33,255 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:25:33,605 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:34,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:35,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:36,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:37,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:38,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:25:38,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:39,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:40,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:41,607 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:42,032 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:25:42,032 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:25:42,607 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:43,607 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:44,063 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:25:44,607 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:45,034 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:25:45,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:25:45,430 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:25:45,430 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:25:45,432 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:25:45,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:46,266 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:25:46,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:47,267 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:25:47,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:48,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:49,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:50,063 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:25:50,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:51,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:52,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:53,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:54,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:55,063 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:25:55,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:56,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:57,032 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:25:57,032 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:25:57,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:58,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:25:58,680 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:25:58,680 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:25:58,682 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:25:58,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:25:59,277 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:25:59,278 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:25:59,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:00,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:26:01,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:02,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:03,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:04,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:05,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:06,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:06,063 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:26:07,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:08,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:09,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:10,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:11,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:11,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:26:11,895 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:26:11,896 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:26:11,897 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:26:11,898 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:26:12,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:12,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:26:12,102 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:26:12,287 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:26:13,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:13,288 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:26:14,215 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:15,038 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:26:15,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:16,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:17,040 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:26:17,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:18,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:19,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:20,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:21,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:22,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:26:22,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:23,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:24,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:24,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:26:24,989 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:26:24,990 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:26:24,991 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:26:25,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:25,298 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:26:26,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:27,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:26:27,097 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:26:27,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:26:27,300 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:26:27,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:28,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:29,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:30,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:31,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:32,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:33,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:26:33,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:34,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:35,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:36,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:37,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:38,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:26:38,231 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:26:38,231 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:26:38,232 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:26:38,233 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:26:38,309 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:26:38,459 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:39,310 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:26:39,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:40,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:41,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:42,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:26:42,097 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:26:42,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:43,373 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:44,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:26:44,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:45,042 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:26:45,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:46,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:47,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:48,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:49,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:26:49,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:50,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:51,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:51,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:26:51,478 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:26:51,478 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:26:51,480 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:26:52,320 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:26:52,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:53,321 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:26:53,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:54,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:55,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:26:55,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:56,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:57,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:26:57,097 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:26:57,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:58,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:26:59,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:00,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:27:00,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:01,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:02,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:03,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:04,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:05,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:06,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:27:06,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:07,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:08,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:09,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:10,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:11,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:12,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:27:12,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:27:12,097 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:27:12,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:13,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:14,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:15,045 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:27:15,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:16,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:17,352 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:27:19,340 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:27:20,032 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:22,434 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:27:28,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:27:33,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:27:34,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:36,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:27:36,002 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:27:36,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:39,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:27:43,821 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:44,165 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:27:45,049 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:27:45,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:27:45,331 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:27:45,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:49,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:27:53,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:27:54,490 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:27:59,581 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:28:00,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:00,974 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 21:28:02,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:28:02,247 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:28:02,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:03,354 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:05,369 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:28:09,896 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 21:28:10,369 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:28:14,184 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:14,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:15,052 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:28:15,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:15,647 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:28:15,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:28:15,842 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:28:16,007 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:28:16,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:17,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:17,649 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:28:18,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:19,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:20,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:21,007 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:28:21,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:22,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:23,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:24,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:25,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:26,008 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:28:26,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:27,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:28,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:29,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:30,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:30,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:28:30,841 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:28:31,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:31,974 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:28:32,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:33,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:34,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:35,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:36,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:36,975 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:28:37,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:38,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:39,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:40,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:41,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:41,976 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:28:42,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:43,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:44,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:45,055 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:28:45,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:45,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:28:45,842 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:28:46,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:47,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:47,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:28:48,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:49,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:50,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:51,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:52,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:52,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:28:53,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:54,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:55,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:56,211 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:57,211 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:57,949 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:28:58,211 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:28:59,211 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:00,211 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:00,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:29:00,842 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:29:01,211 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:02,211 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:02,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:29:03,211 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:04,211 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:05,212 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:06,212 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:07,212 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:07,994 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:29:08,212 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:09,212 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:10,212 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:12,995 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:29:15,059 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:29:18,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:29:23,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:29:28,063 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:29:32,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 21:29:32,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:29:32,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:32,418 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:29:33,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:33,574 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:29:34,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:35,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:36,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:37,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:38,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:38,575 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:29:39,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:40,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:41,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:42,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:43,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:43,575 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:29:44,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:45,063 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:29:45,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:46,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:47,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:29:47,381 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:29:47,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:48,383 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:49,383 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:49,510 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:29:50,383 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:51,383 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:52,383 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:53,383 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:54,383 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:54,510 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:29:55,383 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:56,383 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:57,383 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:58,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:59,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:29:59,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:30:00,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:01,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:02,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:30:02,382 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:30:02,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:03,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:04,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:04,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:30:05,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:06,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:07,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:08,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:09,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:09,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:30:10,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:11,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:12,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:13,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:14,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:14,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:30:15,067 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:30:15,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:16,386 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:17,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:30:17,382 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:30:17,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:18,386 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:19,386 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:20,386 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:20,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:30:21,386 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:22,386 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:23,386 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:24,386 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:25,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:25,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:30:26,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:27,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:28,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:29,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:30,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:30,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:30:31,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:32,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:30:32,381 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:30:32,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:33,389 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:34,389 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:35,389 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:35,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:30:36,391 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:37,391 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:38,391 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:39,391 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:40,392 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:40,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:30:41,392 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:42,392 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:43,392 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:44,392 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:45,070 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:30:45,392 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:46,071 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:30:46,392 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:47,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:30:47,382 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:30:47,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:48,393 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:49,393 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:50,393 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:51,393 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:51,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:30:52,393 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:53,393 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:54,393 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:55,393 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:56,393 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:56,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:30:57,393 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:58,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:30:59,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:00,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:01,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:01,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:31:02,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:31:02,382 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:31:02,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:03,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:04,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:05,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:06,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:07,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:07,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:31:08,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:09,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:10,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:11,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:12,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:12,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:31:13,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:14,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:15,074 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:31:15,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:16,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:17,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:31:17,382 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:31:17,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:17,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:31:18,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:19,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:20,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:21,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:22,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:22,555 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:31:23,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:31:27,556 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:31:32,556 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:31:37,557 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:31:42,558 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:31:45,077 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:31:48,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:31:53,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:31:58,081 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:32:02,861 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 21:32:02,902 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:02,902 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:32:02,903 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:32:03,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:03,995 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:32:04,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:05,921 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:06,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:08,050 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:08,389 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:32:09,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:09,867 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:32:10,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:11,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:12,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:13,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:14,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:14,868 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:32:15,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:15,080 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:32:16,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:17,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:17,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:32:17,863 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:32:18,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:19,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:19,966 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:32:20,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:32:20,036 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:32:20,036 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:32:20,038 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:32:20,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:20,399 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:32:21,184 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:22,184 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:22,401 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:32:23,184 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:24,184 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:25,076 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:32:25,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:26,184 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:27,184 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:28,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:29,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:30,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:32:30,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:31,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:32,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:32,862 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:32:32,863 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:32:33,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:33,811 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:32:33,813 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:32:33,813 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:32:33,815 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:32:34,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:34,409 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:32:35,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:35,853 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:32:36,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:36,411 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:32:37,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:38,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:39,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:40,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:40,854 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:32:41,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:42,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:43,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:44,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:45,084 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:32:45,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:46,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:32:46,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:47,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:47,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:32:47,624 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:32:47,625 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:32:47,626 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:32:47,862 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:32:47,863 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:32:48,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:48,420 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:32:49,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:50,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:50,422 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:32:51,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:52,007 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:32:52,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:53,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:54,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:55,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:56,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:57,008 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:32:57,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:58,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:32:59,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:00,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:01,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:01,324 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:33:01,326 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:33:01,326 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:33:01,328 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:33:01,430 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:33:02,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:02,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:33:02,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:33:02,863 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:33:03,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:04,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:04,433 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:33:05,191 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:06,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:07,191 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:07,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:33:08,191 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:09,191 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:10,191 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:11,191 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:12,191 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:12,984 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:33:13,191 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:14,191 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:15,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:33:15,043 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:33:15,044 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:33:15,045 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:33:15,087 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:33:15,260 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:15,441 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:33:16,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:16,442 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:33:17,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:17,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:33:17,863 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:33:18,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:33:18,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:19,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:20,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:21,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:22,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:23,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:33:23,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:24,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:25,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:26,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:27,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:28,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:33:28,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:28,638 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:33:28,640 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:33:28,641 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:33:28,642 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:33:29,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:29,452 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:33:30,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:30,453 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:33:31,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:32,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:32,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:33:32,863 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:33:33,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:34,009 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:33:34,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:35,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:36,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:37,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:38,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:39,010 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:33:39,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:40,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:41,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:42,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:33:42,308 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:33:42,309 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:33:42,310 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:33:42,461 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:33:42,534 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:43,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:44,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:33:44,463 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:33:44,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:45,091 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:33:45,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:46,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:47,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:47,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:33:47,864 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:33:48,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:49,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:49,966 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:33:50,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:51,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:52,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:53,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:54,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:54,967 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:33:55,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:56,130 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:33:56,132 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:33:56,133 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:33:56,134 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:33:56,473 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:33:56,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:57,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:58,475 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:33:58,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:33:59,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:00,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:34:00,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:01,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:02,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:02,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:34:02,864 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:34:03,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:04,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:05,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:06,005 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:34:06,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:07,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:08,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:09,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:09,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:34:09,858 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:34:09,859 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:34:09,860 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:34:10,484 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:34:10,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:11,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:11,898 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:34:12,485 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:34:12,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:13,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:14,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:15,096 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:34:15,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:16,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:17,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:34:17,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:17,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:34:17,864 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:34:18,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:19,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:20,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:21,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:22,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:23,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:34:23,587 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:34:23,588 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:34:23,589 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:34:23,590 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:34:23,790 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:24,495 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:34:24,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:25,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:26,497 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:34:26,726 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:27,726 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:28,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:34:28,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:29,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:30,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:31,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:32,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:32,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:34:32,864 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:34:33,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:33,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:34:34,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:35,757 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:36,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:37,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:34:37,399 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:34:37,399 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:34:37,401 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:34:37,505 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:34:37,757 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:38,757 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:39,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:34:39,757 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:40,507 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:34:40,757 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:41,757 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:42,757 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:43,757 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:44,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:34:44,757 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:45,099 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:34:45,758 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:46,758 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:47,758 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:47,864 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:34:47,864 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:34:48,758 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:49,758 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:50,008 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:34:50,758 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:51,120 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:34:51,121 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:34:51,122 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:34:51,124 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:34:51,515 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:34:51,758 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:52,517 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:34:52,758 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:53,759 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:54,895 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:55,162 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:34:55,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:56,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:57,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:58,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:34:59,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:00,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:35:00,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:01,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:02,864 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:35:02,864 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:35:02,905 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:03,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:04,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:04,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:35:04,942 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:35:04,942 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:35:04,944 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:35:05,526 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:35:05,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:05,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:35:06,527 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:35:06,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:07,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:08,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:10,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:10,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:35:11,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:12,002 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:13,002 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:14,002 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:15,002 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:15,103 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:35:16,002 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:16,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:35:17,002 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:17,864 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:35:17,864 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:35:18,002 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:18,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:35:18,635 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:35:18,636 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:35:18,637 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:35:19,002 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:19,537 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:35:20,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:20,538 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:35:21,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:21,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:35:22,005 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:23,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:24,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:25,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:26,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:26,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:35:27,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:28,004 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:29,004 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:30,004 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:31,004 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:31,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:35:32,004 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:32,392 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:35:32,394 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:35:32,394 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:35:32,395 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:35:32,547 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:35:32,864 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:35:32,865 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:35:33,005 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:34,004 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:34,548 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:35:35,005 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:36,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:37,009 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:35:37,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:38,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:39,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:40,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:41,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:42,010 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:35:42,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:43,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:44,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:45,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:45,106 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:35:46,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:46,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:35:46,166 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:35:46,166 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:35:46,168 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:35:46,557 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:35:47,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:47,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:35:47,864 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:35:47,865 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:35:48,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:48,559 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:35:49,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:50,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:51,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:52,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:53,002 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:35:53,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:54,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:55,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:56,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:57,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:58,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:35:58,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:59,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:35:59,926 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:35:59,928 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:35:59,928 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:35:59,930 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:36:00,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:00,567 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:36:01,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:02,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:02,569 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:36:02,864 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:36:02,865 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:36:03,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:03,974 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:36:04,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:05,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:06,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:07,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:08,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:08,975 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:36:09,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:10,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:11,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:12,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:13,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:13,618 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:36:13,620 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:36:13,620 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:36:13,621 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:36:14,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:14,577 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:36:14,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:36:15,110 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:36:15,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:16,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:16,580 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:36:17,378 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:17,864 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:36:17,865 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:36:18,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:19,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:19,969 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:36:20,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:21,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:22,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:23,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:24,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:24,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:36:25,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:26,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:27,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:36:27,274 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:36:27,275 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:36:27,277 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:36:27,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:27,588 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:36:28,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:28,589 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:36:29,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:30,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:36:30,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:31,409 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:32,456 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:32,865 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:36:32,865 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:36:33,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:34,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:35,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:35,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:36:36,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:37,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:38,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:39,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:40,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:40,899 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:36:40,901 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:36:40,901 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:36:40,903 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:36:40,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:36:41,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:41,598 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:36:42,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:42,599 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:36:43,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:44,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:45,115 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:36:45,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:46,117 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:36:46,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:47,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:47,865 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:36:47,865 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:36:48,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:49,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:50,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:51,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:52,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:36:52,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:53,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:54,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:54,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:36:54,625 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:36:54,626 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:36:54,628 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:36:55,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:55,608 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:36:56,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:56,609 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:36:57,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:57,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:36:58,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:36:59,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:00,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:01,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:02,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:02,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:37:02,865 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:37:02,865 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:37:03,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:04,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:05,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:06,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:07,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:08,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:37:08,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:37:08,190 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:37:08,191 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:37:08,193 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:37:08,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:08,617 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:37:09,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:10,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:10,619 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:37:11,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:12,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:13,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:37:13,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:14,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:15,117 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:37:15,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:16,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:17,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:17,865 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:37:17,865 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:37:18,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:19,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:37:19,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:20,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:21,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:21,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:37:21,766 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:37:21,767 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:37:21,768 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:37:22,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:22,628 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:37:22,629 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:37:23,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:24,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:24,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:37:25,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:26,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:27,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:28,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:29,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:29,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:37:30,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:31,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:32,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:32,865 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:37:32,866 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:37:33,506 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:34,506 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:35,033 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:37:35,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:37:35,345 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:37:35,346 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:37:35,347 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:37:35,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:35,638 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:37:36,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:36,639 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:37:37,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:38,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:39,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:40,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:37:40,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:41,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:42,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:43,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:44,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:45,121 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:37:45,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:46,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:37:46,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:47,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:47,865 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:37:47,866 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:37:48,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:48,963 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:37:48,965 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:37:48,965 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:37:48,967 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:37:49,648 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:37:49,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:50,650 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:37:50,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:51,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:52,005 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:37:52,683 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:53,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:54,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:55,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:56,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:57,005 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:37:57,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:58,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:37:59,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:00,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:01,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:02,006 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:38:02,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:02,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:38:02,737 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:38:02,738 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:38:02,739 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:38:02,953 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:38:02,954 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:38:03,658 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:38:03,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:04,660 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:38:04,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:05,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:06,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:07,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:38:07,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:08,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:09,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:10,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:11,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:12,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:38:12,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:13,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:14,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:15,125 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:38:15,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:16,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:38:16,379 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:38:16,379 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:38:16,381 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:38:16,668 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:38:16,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:17,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:38:17,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:17,871 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:38:17,872 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:38:18,670 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:38:18,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:19,698 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:20,698 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:21,698 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:22,698 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:22,965 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:38:23,698 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:24,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:25,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:26,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:27,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:27,966 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:38:28,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:29,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:30,008 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:38:30,010 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:38:30,010 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:38:30,012 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:38:30,679 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:38:30,679 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:38:30,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:31,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:32,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:32,871 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:38:32,871 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:38:33,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:38:33,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:34,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:35,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:36,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:37,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:38,043 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:38:38,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:39,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:40,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:41,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:42,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:43,044 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:38:43,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:38:43,634 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:38:43,634 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:38:43,636 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:38:43,687 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:38:43,862 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:44,689 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:38:44,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:45,128 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:38:45,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:46,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:47,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:47,871 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:38:47,871 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:38:48,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:49,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:38:49,786 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:50,786 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:51,786 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:52,786 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:53,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:54,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:38:54,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:55,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:56,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:57,343 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:38:57,345 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:38:57,345 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:38:57,347 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:38:57,698 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:38:57,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:58,699 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:38:58,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:38:59,386 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:38:59,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:00,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:01,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:02,789 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:02,871 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:39:02,872 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:39:03,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:04,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:05,004 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:39:05,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:06,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:07,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:08,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:09,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:10,005 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:39:10,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:10,997 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:39:10,998 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:39:10,999 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:39:11,000 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:39:11,708 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:39:11,789 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:12,709 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:39:12,789 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:13,789 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:14,789 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:15,038 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:39:15,132 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:39:15,789 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:16,789 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:17,789 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:17,871 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:39:17,872 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:39:18,790 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:19,790 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:20,790 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:20,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:39:21,790 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:22,790 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:23,790 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:24,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:39:24,626 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:39:24,627 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:39:24,628 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:39:24,718 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:39:24,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:25,855 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:26,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:39:26,720 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:39:26,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:27,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:28,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:29,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:30,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:31,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:39:31,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:32,871 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:39:32,872 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:39:32,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:33,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:34,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:35,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:36,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:36,945 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:39:37,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:38,235 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:39:38,237 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:39:38,237 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:39:38,239 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:39:38,728 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:39:38,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:39,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:40,730 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:39:40,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:41,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:42,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:39:42,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:43,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:44,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:45,136 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:39:45,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:46,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:47,871 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:39:47,872 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:39:47,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:48,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:39:48,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:49,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:50,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:51,801 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:39:51,803 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:39:51,804 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:39:51,805 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:39:52,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:52,740 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:39:52,740 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:39:53,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:53,843 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:39:54,129 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:55,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:56,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:57,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:58,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:39:58,844 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:39:59,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:00,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:01,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:02,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:02,872 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:40:02,872 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:40:03,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:04,039 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:40:04,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:05,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:05,569 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:40:05,571 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:40:05,572 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:40:05,573 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:40:05,749 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:40:06,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:06,750 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:40:07,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:08,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:09,254 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:09,611 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:40:10,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:11,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:12,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:13,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:14,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:14,612 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:40:15,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:15,139 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:40:16,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:17,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:17,872 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:40:17,872 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:40:18,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:19,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:19,317 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:40:19,319 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:40:19,319 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:40:19,320 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:40:19,760 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:40:20,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:20,359 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:40:20,762 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:40:21,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:22,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:23,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:24,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:25,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:25,359 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:40:26,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:27,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:28,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:29,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:30,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:30,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:40:31,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:32,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:32,872 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:40:32,872 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:40:33,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:40:33,041 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:40:33,042 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:40:33,043 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:40:33,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:33,772 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:40:34,149 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:34,773 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:40:35,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:36,044 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:40:36,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:37,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:38,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:39,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:40,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:41,045 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:40:41,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:42,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:43,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:44,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:45,144 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:40:45,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:46,146 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:40:46,165 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:46,954 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:40:46,956 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:40:46,956 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:40:46,958 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:40:47,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:47,783 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:40:47,872 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:40:47,873 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:40:48,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:48,784 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:40:49,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:50,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:51,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:51,969 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:40:52,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:53,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:54,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:55,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:56,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:56,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:40:57,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:58,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:40:59,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:00,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:00,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:41:00,687 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:41:00,687 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:41:00,689 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:41:00,792 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:41:01,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:02,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:02,728 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:41:02,794 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:41:03,006 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:41:03,007 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:41:03,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:04,389 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:05,378 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:06,378 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:07,378 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:08,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:41:08,378 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:09,378 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:10,379 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:11,379 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:12,379 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:13,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:41:13,379 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:14,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:41:14,337 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:41:14,338 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:41:14,340 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:41:14,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:14,803 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:41:14,804 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:41:15,147 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:41:15,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:16,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:17,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:18,060 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:41:18,060 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:41:18,221 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:41:18,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:19,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:20,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:21,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:22,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:23,221 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:41:23,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:24,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:25,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:26,670 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:27,670 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:28,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:41:28,019 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:41:28,019 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:41:28,020 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:41:28,670 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:28,815 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:41:28,815 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:41:29,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:41:29,670 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:30,670 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:31,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:32,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:33,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:41:33,170 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:41:33,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:34,253 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:41:34,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:35,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:36,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:37,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:38,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:39,254 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:41:39,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:40,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:41,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:41:41,659 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:41:41,660 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:41:41,662 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:41:41,825 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:41:41,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:42,826 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:41:42,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:43,978 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:44,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:41:44,977 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:45,150 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:41:45,977 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:46,978 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:47,978 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:48,157 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:41:48,158 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:41:48,978 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:49,978 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:50,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:41:50,978 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:51,978 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:52,978 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:53,978 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:54,978 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:55,139 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:41:55,140 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:41:55,141 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:41:55,142 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:41:55,839 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:41:55,978 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:56,180 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:41:56,840 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:41:56,979 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:57,979 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:58,979 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:41:59,980 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:00,979 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:01,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:42:01,979 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:02,979 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:03,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:42:03,158 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:42:03,979 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:04,979 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:05,980 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:06,236 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:42:06,980 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:07,980 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:08,652 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:42:08,654 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:42:08,655 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:42:08,656 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:42:08,849 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:42:08,980 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:09,980 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:10,851 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:42:10,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:11,694 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:42:11,980 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:12,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:13,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:14,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:15,154 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:42:15,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:16,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:17,157 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:42:17,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:18,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:42:18,158 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:42:18,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:19,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:20,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:21,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:22,139 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:42:22,140 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:42:22,141 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:42:22,142 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:42:22,180 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:42:22,861 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:42:22,861 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:42:22,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:23,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:24,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:25,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:26,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:27,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:42:27,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:28,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:29,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:30,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:31,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:32,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:42:32,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:33,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:42:33,159 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:42:33,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:34,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:35,570 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:42:35,572 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:42:35,572 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:42:35,574 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:42:35,870 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:42:35,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:36,871 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:42:36,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:37,613 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:42:37,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:38,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:39,984 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:40,984 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:41,984 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:42,613 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:42:42,984 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:43,984 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:44,984 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:45,158 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:42:45,984 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:46,984 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:47,984 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:48,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:42:48,159 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:42:48,304 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:42:48,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:49,043 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:42:49,044 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:42:49,045 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:42:49,046 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:42:49,881 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:42:49,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:50,882 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:42:50,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:51,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:52,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:53,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:54,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:42:54,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:55,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:56,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:57,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:58,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:42:59,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:42:59,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:00,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:01,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:02,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:43:02,601 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:43:02,602 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:43:02,603 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:43:02,891 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:43:02,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:03,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:43:03,159 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:43:03,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:04,324 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:43:04,892 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:43:04,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:05,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:06,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:07,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:08,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:09,325 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:43:09,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:10,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:11,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:12,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:13,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:14,326 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:43:14,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:15,162 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:43:15,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:16,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:43:16,069 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:43:16,069 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:43:16,071 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:43:16,902 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:43:16,902 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:43:16,988 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:17,988 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:18,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:43:18,159 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:43:18,988 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:19,988 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:20,320 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:43:20,988 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:21,988 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:22,988 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:23,988 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:24,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:25,320 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:43:25,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:26,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:27,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:28,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:29,576 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:43:29,578 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:43:29,579 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:43:29,580 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:43:29,911 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:43:29,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:30,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:43:30,912 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:43:30,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:31,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:32,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:33,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:43:33,275 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:43:33,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:34,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:35,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:36,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:43:36,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:37,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:38,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:39,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:40,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:41,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:43:41,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:42,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:43,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:43:43,029 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:43:43,030 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:43:43,031 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:43:43,922 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:43:43,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:44,923 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:43:44,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:45,165 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:43:45,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:46,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:47,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:43:47,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:48,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:43:48,274 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:43:48,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:49,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:50,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:51,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:52,401 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:43:52,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:53,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:54,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:55,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:56,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:43:56,663 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:43:56,664 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:43:56,665 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:43:56,932 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:43:56,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:57,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:43:57,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:58,934 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:43:59,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:43:59,997 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:00,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:01,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:02,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:44:02,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:03,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:44:03,275 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:44:03,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:04,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:05,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:06,999 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:07,999 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:08,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:44:08,999 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:09,999 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:10,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:44:10,153 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:44:10,154 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:44:10,155 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:44:10,943 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:44:10,944 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:44:10,999 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:11,999 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:13,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:14,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:14,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:44:15,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:15,168 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:44:16,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:17,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:18,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:18,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:44:18,275 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:44:19,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:19,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:44:20,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:21,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:22,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:23,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:23,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:44:23,741 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:44:23,741 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:44:23,743 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:44:23,954 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:44:24,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:24,781 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:44:24,956 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:44:25,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:26,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:27,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:28,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:29,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:29,782 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:44:30,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:31,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:32,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:33,063 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:33,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:44:33,275 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:44:34,063 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:35,063 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:35,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:44:36,063 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:37,063 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:37,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:44:37,211 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:44:37,212 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:44:37,213 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:44:37,967 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:44:38,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:38,968 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:44:39,063 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:40,063 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:41,063 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:41,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:44:42,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:43,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:44,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:45,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:45,172 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:44:46,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:47,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:47,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:44:48,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:48,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:44:48,275 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:44:49,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:50,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:50,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:44:50,661 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:44:50,662 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:44:50,664 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:44:50,975 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:44:51,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:52,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:52,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:44:52,977 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:44:53,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:54,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:55,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:56,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:57,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:57,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:44:58,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:44:59,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:00,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:01,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:02,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:02,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:45:03,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:03,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:45:03,275 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:45:04,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:04,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:45:04,106 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:45:04,106 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:45:04,108 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:45:04,986 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:45:04,986 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:45:05,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:06,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:07,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:08,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:08,146 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:45:09,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:10,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:11,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:12,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:13,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:13,147 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:45:14,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:15,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:15,174 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:45:16,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:17,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:17,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:45:17,637 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:45:17,637 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:45:17,639 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:45:17,994 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:45:18,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:18,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:45:18,275 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:45:18,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:45:18,996 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:45:19,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:20,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:21,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:22,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:23,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:23,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:45:24,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:25,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:26,069 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:27,069 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:28,069 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:28,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:45:29,069 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:30,069 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:31,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:45:31,025 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:45:31,026 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:45:31,027 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:45:31,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:32,004 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:45:32,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:33,005 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:45:33,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:33,338 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:45:33,338 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:45:33,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:45:34,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:35,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:36,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:37,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:38,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:38,480 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:45:39,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:40,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:41,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:42,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:43,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:43,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:45:44,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:44,581 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:45:44,582 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:45:44,583 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:45:44,584 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:45:45,016 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:45:45,017 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:45:45,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:45,178 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:45:46,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:47,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:48,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:48,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:45:48,335 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:45:49,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:49,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:45:50,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:51,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:52,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:53,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:54,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:54,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:45:55,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:56,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:57,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:58,149 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:45:58,151 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:45:58,152 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:45:58,153 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:45:58,370 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:45:59,026 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:45:59,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:00,027 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:46:00,191 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:46:00,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:01,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:02,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:03,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:03,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:46:03,333 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:46:04,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:05,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:05,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:46:06,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:07,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:08,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:09,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:10,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:10,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:46:11,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:11,758 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:46:11,760 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:46:11,760 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:46:11,762 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:46:12,035 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:46:12,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:13,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:14,037 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:46:14,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:15,180 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:46:15,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:16,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:46:16,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:17,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:18,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:18,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:46:18,333 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:46:19,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:20,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:21,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:21,472 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:46:22,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:23,290 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:24,290 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:25,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:46:25,190 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:46:25,191 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:46:25,192 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:46:25,410 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:26,046 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:46:26,409 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:27,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:46:27,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:28,048 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:46:28,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:29,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:30,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:31,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:32,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:46:32,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:33,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:46:33,333 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:46:33,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:34,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:35,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:36,513 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:37,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:46:37,513 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:38,513 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:38,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:46:38,621 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:46:38,622 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:46:38,623 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:46:39,055 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:46:39,513 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:40,056 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:46:40,513 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:41,513 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:42,513 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:42,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:46:43,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:44,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:45,182 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:46:45,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:46,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:47,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:48,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:46:48,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:46:48,333 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:46:48,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:49,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:50,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:51,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:51,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:46:51,893 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:46:51,893 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:46:51,895 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:46:52,065 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:46:52,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:53,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:53,933 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:46:54,066 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:46:54,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:55,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:56,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:57,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:58,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:46:58,933 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:46:59,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:00,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:01,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:02,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:03,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:47:03,334 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:47:03,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:04,483 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:47:04,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:05,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:47:05,360 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:47:05,360 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:47:05,361 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:47:05,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:06,075 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:47:06,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:07,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:08,076 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:47:08,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:09,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:10,400 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:47:10,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:11,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:12,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:13,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:14,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:15,186 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:47:15,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:16,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:47:16,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:17,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:18,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:47:18,334 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:47:18,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:18,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:47:18,740 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:47:18,741 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:47:18,742 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:47:19,084 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:47:19,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:20,085 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:47:20,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:21,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:21,781 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:47:22,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:23,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:24,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:25,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:26,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:26,782 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:47:27,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:28,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:29,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:30,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:31,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:31,782 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:47:32,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:47:32,229 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:47:32,229 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:47:32,231 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:47:32,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:33,094 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:47:33,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:47:33,335 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:47:33,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:34,095 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:47:34,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:35,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:36,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:37,457 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:47:37,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:38,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:39,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:40,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:41,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:42,458 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:47:42,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:43,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:44,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:45,188 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:47:45,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:45,728 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:47:45,729 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:47:45,730 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:47:45,731 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:47:46,104 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:47:46,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:47,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:47,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:47:48,106 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:47:48,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:47:48,334 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:47:48,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:49,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:50,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:51,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:52,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:53,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:47:53,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:54,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:55,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:56,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:57,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:58,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:47:58,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:47:59,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:47:59,332 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:47:59,333 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:47:59,334 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:47:59,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:00,114 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:48:00,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:01,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:02,115 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:48:02,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:03,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:48:03,335 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:48:03,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:04,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:48:04,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:05,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:06,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:07,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:08,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:09,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:48:09,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:10,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:11,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:12,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:12,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:48:12,788 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:48:12,789 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:48:12,790 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:48:13,124 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:48:13,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:14,125 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:48:14,680 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:14,829 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:48:15,191 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:48:15,680 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:16,680 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:17,680 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:18,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:48:18,335 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:48:18,680 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:19,680 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:20,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:48:20,680 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:21,681 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:22,681 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:23,681 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:24,681 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:25,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:48:25,681 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:26,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:48:26,257 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:48:26,257 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:48:26,259 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:48:26,681 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:27,134 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:48:27,682 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:28,135 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:48:28,682 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:29,682 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:30,682 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:31,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:48:31,682 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:32,683 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:33,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:48:33,335 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:48:33,683 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:34,683 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:35,683 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:36,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:48:36,683 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:37,683 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:38,683 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:39,683 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:39,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:48:39,746 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:48:39,746 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:48:39,748 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:48:40,143 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:48:40,684 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:41,684 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:41,786 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:48:42,145 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:48:42,684 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:43,684 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:44,684 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:45,194 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:48:45,684 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:46,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:47,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:48:47,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:48,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:48:48,335 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:48:48,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:49,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:50,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:51,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:52,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:48:52,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:53,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:48:53,189 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:48:53,190 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:48:53,192 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:48:53,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:54,152 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:48:54,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:55,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:56,154 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:48:56,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:57,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:58,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:48:58,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:48:59,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:00,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:01,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:02,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:03,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:49:03,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:49:03,335 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:49:03,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:04,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:05,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:06,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:49:06,677 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:49:06,677 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:49:06,679 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:49:06,892 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:07,162 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:49:07,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:08,163 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:49:08,717 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:49:08,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:09,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:10,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:11,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:12,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:13,717 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:49:13,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:14,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:15,198 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:49:15,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:16,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:17,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:18,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:49:18,335 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:49:18,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:19,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:49:19,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:20,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:49:20,057 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:49:20,058 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:49:20,059 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:49:20,173 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:49:20,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:21,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:22,175 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:49:22,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:23,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:24,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:25,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:49:25,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:26,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:27,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:28,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:29,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:30,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:49:30,820 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:31,820 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:32,820 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:33,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:49:33,335 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:49:33,570 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:49:33,572 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:49:33,572 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:49:33,574 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:49:33,820 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:34,183 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:49:34,820 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:35,575 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:49:35,893 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:36,185 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:49:36,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:37,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:38,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:39,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:40,576 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:49:40,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:41,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:42,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:43,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:44,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:45,202 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:49:45,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:46,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:49:46,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:47,037 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:49:47,038 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:49:47,039 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:49:47,040 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:49:47,192 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:49:47,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:48,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:49:48,335 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:49:48,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:49,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:50,198 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:49:50,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:51,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:49:51,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:52,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:53,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:54,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:55,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:56,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:49:56,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:57,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:58,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:49:59,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:00,418 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:50:00,419 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:50:00,420 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:50:00,422 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:50:00,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:01,205 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:50:01,459 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:50:01,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:02,206 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:50:02,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:03,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:50:03,336 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:50:03,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:04,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:05,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:06,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:50:06,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:07,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:08,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:09,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:10,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:11,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:50:11,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:12,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:13,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:13,956 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:50:13,958 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:50:13,958 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:50:13,960 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:50:14,214 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:50:14,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:15,206 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:50:15,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:16,216 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:50:16,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:17,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:50:17,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:18,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:50:18,336 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:50:18,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:19,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:20,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:21,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:22,437 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:50:22,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:23,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:24,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:25,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:26,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:27,437 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:50:27,463 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:50:27,465 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:50:27,465 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:50:27,467 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:50:28,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:28,224 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:50:29,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:30,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:30,226 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:50:31,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:32,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:32,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:50:33,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:33,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:50:33,336 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:50:34,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:35,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:36,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:37,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:38,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:38,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:50:39,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:40,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:41,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:41,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:50:41,196 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:50:41,196 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:50:41,198 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:50:41,233 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:50:42,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:43,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:44,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:44,235 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:50:44,236 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:50:45,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:45,209 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:50:46,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:47,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:48,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:48,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:50:48,336 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:50:49,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:49,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:50:50,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:51,093 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:52,093 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:53,093 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:54,093 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:54,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:50:54,603 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:50:54,604 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:50:54,605 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:50:54,606 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:50:55,093 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:55,243 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:50:56,094 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:56,244 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:50:57,094 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:58,094 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:59,094 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:50:59,644 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:51:00,094 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:01,094 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:02,094 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:03,094 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:03,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:51:03,336 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:51:04,095 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:05,095 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:05,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:51:06,095 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:07,095 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:07,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:51:07,983 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:51:07,984 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:51:07,986 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:51:08,215 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:08,252 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:51:09,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:10,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:10,254 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:51:11,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:51:11,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:12,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:13,126 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:14,125 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:15,125 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:15,213 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:51:16,126 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:16,215 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:51:17,126 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:18,126 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:18,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:51:18,336 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:51:19,126 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:20,126 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:21,126 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:21,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:51:21,373 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:51:21,373 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:51:21,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:51:21,375 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:51:22,126 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:22,263 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:51:23,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:24,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:24,265 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:51:25,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:26,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:26,413 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:51:27,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:28,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:29,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:30,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:31,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:31,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:51:32,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:33,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:33,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:51:33,336 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:51:34,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:34,820 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:51:34,822 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:51:34,822 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:51:34,824 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:51:35,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:35,275 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:51:36,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:36,275 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:51:36,862 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:51:37,139 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:38,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:39,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:40,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:41,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:41,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:51:42,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:43,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:44,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:45,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:45,217 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:51:46,139 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:47,139 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:47,219 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:51:48,139 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:48,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:51:48,179 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:51:48,179 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:51:48,181 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:51:48,284 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:51:48,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:51:48,397 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:51:49,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:50,139 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:50,286 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:51:51,139 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:52,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:52,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:51:53,139 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:54,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:55,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:56,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:57,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:57,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:51:58,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:51:59,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:00,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:01,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:01,608 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:52:01,609 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:52:01,610 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:52:01,612 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:52:02,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:02,294 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:52:02,651 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:52:03,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:03,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:52:03,395 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:52:04,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:04,296 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:52:05,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:06,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:07,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:08,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:08,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:52:09,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:10,142 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:11,142 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:12,142 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:13,142 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:13,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:52:14,142 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:15,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:52:15,087 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:52:15,087 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:52:15,089 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:52:15,220 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:52:15,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:15,304 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:52:16,217 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:16,305 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:52:17,218 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:18,217 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:18,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:52:18,395 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:52:18,527 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:52:19,217 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:20,236 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:21,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:22,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:23,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:23,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:52:24,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:25,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:26,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:27,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:28,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:28,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:52:28,569 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:52:28,571 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:52:28,571 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:52:28,573 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:52:29,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:29,314 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:52:30,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:30,315 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:52:31,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:32,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:33,260 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:33,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:52:33,396 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:52:34,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:34,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:52:35,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:36,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:37,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:38,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:39,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:39,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:52:40,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:41,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:42,060 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:52:42,061 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:52:42,062 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:52:42,063 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:52:42,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:42,323 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:52:43,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:44,325 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:52:44,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:45,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:52:45,224 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:52:45,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:46,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:47,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:48,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:48,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:52:48,396 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:52:49,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:50,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:50,559 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:52:51,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:52,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:53,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:54,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:55,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:55,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:52:55,681 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:52:55,682 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:52:55,683 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:52:55,684 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:52:56,334 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:52:56,390 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:57,390 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:58,336 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:52:58,390 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:52:59,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:00,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:00,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:53:01,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:02,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:03,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:53:03,396 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:53:03,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:04,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:05,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:06,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:06,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:53:07,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:08,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:09,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:53:09,270 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:53:09,271 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:53:09,272 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:53:09,344 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:53:09,482 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:10,345 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:53:10,482 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:11,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:12,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:53:12,581 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:13,582 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:14,582 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:15,227 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:53:15,582 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:16,582 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:17,582 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:18,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:53:18,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:53:18,396 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:53:18,582 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:19,582 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:20,582 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:21,582 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:22,582 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:22,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:53:22,712 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:53:22,713 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:53:22,714 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:53:23,354 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:53:23,583 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:23,753 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:53:24,354 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:53:24,583 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:25,583 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:26,583 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:27,583 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:28,583 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:28,753 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:53:29,583 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:30,583 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:31,583 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:32,583 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:33,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:53:33,397 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:53:33,584 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:34,472 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:53:34,584 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:35,584 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:36,094 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:53:36,096 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:53:36,097 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:53:36,098 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:53:36,362 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:53:36,584 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:37,584 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:38,364 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:53:38,584 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:39,584 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:40,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:53:40,584 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:41,585 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:42,584 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:43,585 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:44,585 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:45,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:53:45,231 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:53:45,585 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:46,585 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:47,585 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:48,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:53:48,397 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:53:48,585 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:49,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:53:49,488 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:53:49,489 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:53:49,490 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:53:49,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:50,372 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:53:50,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:53:50,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:51,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:52,373 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:53:52,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:53,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:54,638 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:55,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:53:55,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:56,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:57,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:58,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:53:59,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:00,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:54:00,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:01,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:02,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:02,979 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:54:02,980 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:54:02,981 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:54:02,983 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:54:03,381 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:54:03,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:54:03,396 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:54:03,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:04,382 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:54:04,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:05,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:06,478 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:54:06,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:07,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:08,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:09,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:10,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:11,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:54:11,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:12,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:13,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:14,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:15,235 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:54:15,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:16,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:54:16,449 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:54:16,450 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:54:16,451 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:54:16,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:54:16,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:17,390 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:54:17,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:18,390 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:54:18,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:54:18,396 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:54:18,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:19,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:20,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:21,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:54:21,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:22,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:23,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:24,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:25,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:26,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:54:26,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:27,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:28,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:29,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:29,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:54:29,758 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:54:29,758 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:54:29,760 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:54:30,399 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:54:30,400 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:54:30,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:31,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:31,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:54:32,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:33,463 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:54:33,464 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:54:33,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:34,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:35,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:36,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:37,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:54:37,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:38,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:39,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:40,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:41,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:42,574 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:54:42,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:43,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 21:54:43,241 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 21:54:43,242 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 21:54:43,244 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 21:54:43,409 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 21:54:43,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:44,410 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:54:44,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:45,238 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:54:45,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:46,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:47,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:48,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:54:48,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:54:48,453 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:54:48,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:49,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:50,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:51,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:52,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:53,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:54:53,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:54,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:55,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:56,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:57,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:58,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:54:58,728 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:54:59,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:00,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:01,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:02,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:03,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:55:03,454 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:55:03,584 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:55:03,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:04,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:05,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:06,728 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:07,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:08,584 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:55:08,828 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:10,428 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:55:10,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:13,870 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:55:15,245 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:55:17,844 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 21:55:18,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:19,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:55:20,095 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:55:20,095 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:55:20,144 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:25,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:55:26,061 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 21:55:27,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:28,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:30,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:55:35,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:55:36,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:55:36,093 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:55:36,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:37,339 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:40,192 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:55:43,828 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 21:55:45,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:55:45,245 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:55:45,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:50,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:55:52,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 21:55:53,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:55:54,040 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:55:54,041 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:55:56,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:56:00,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 21:56:01,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:56:05,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:05,558 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:06,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:06,559 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:56:06,836 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:56:07,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:56:07,019 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:56:07,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:08,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:08,838 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:56:09,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:10,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:11,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:12,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:56:12,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:13,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:14,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:15,249 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:56:15,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:16,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:17,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:56:17,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:18,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:19,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:20,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:21,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:22,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:56:22,019 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:56:22,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:23,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:56:23,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:24,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:25,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:26,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:27,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:28,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:56:28,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:29,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:30,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:31,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:32,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:33,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:56:33,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:34,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:35,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:36,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:37,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:56:37,019 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:56:37,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:38,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:56:38,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:39,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:40,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:41,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:42,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:43,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:56:43,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:44,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:45,257 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:56:45,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:46,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:47,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:48,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:56:48,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:49,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:50,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:51,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:52,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:56:52,019 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:56:52,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:53,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:54,192 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:56:54,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:55,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:56,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:57,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:58,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:56:59,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:56:59,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:04,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:57:09,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:57:14,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:57:15,256 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:57:19,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:57:21,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 21:57:21,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:57:21,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:21,142 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:57:22,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:23,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:24,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:24,264 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:57:25,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:26,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:27,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:28,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:29,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:29,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:57:30,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:31,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:32,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:33,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:34,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:34,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:57:35,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:36,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:57:36,096 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:57:36,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:37,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:38,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:39,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:40,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:40,222 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:57:41,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:42,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:43,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:44,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:45,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:45,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:57:45,259 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:57:46,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:47,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:48,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:49,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:50,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:50,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:57:51,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:57:51,096 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:57:51,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:52,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:53,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:54,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:55,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:56,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:56,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:57:57,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:58,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:57:59,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:00,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:01,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:01,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:58:02,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:03,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:04,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:05,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:06,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:58:06,096 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:58:06,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:06,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:58:07,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:08,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:09,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:10,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:11,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:11,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:58:12,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:13,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:14,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:15,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:15,263 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:58:16,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:17,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:17,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:58:18,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:19,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:20,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:21,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:58:21,097 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:58:21,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:22,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:23,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:23,236 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:58:24,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:25,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:26,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:27,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:28,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:28,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:58:29,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:30,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:31,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:32,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:33,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:33,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:58:34,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:35,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:36,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:58:36,097 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:58:36,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:37,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:38,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:38,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:58:39,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:40,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:41,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:42,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:43,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:43,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:58:44,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:45,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:45,301 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:58:46,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:47,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:48,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:48,304 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:58:49,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:50,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:51,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:58:51,097 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:58:51,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:52,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:53,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:54,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:54,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:58:55,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:56,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:57,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:58,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:59,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:58:59,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:59:00,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:01,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:02,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:03,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:04,111 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:04,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:59:09,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:59:14,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:59:15,270 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:59:19,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:59:24,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:59:29,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:59:34,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:59:39,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:59:44,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:59:45,278 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 21:59:47,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 21:59:47,457 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:47,457 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 21:59:47,458 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 21:59:48,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:49,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:49,559 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:59:50,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:50,636 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:59:51,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:52,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:52,638 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 21:59:53,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:54,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:54,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 21:59:55,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:56,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:57,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:58,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:59,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 21:59:59,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:00:00,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:01,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:02,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:00:02,421 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:00:02,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:03,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:03,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:00:03,714 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:00:03,715 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:00:03,716 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:00:04,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:04,645 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:00:04,645 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:00:04,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:00:05,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:06,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:07,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:08,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:09,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:09,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:00:10,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:11,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:12,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:13,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:14,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:14,755 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:00:15,277 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:00:15,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:16,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:17,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:00:17,421 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:00:17,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:17,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:00:17,515 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:00:17,515 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:00:17,517 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:00:17,653 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:00:18,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:18,654 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:00:19,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:20,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:20,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:00:21,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:22,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:23,540 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:24,540 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:25,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:00:25,540 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:26,540 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:27,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:28,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:29,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:30,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:00:30,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:31,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:00:31,107 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:00:31,107 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:00:31,109 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:00:31,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:31,663 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:00:32,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:00:32,421 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:00:32,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:32,664 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:00:33,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:34,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:35,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:00:35,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:36,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:37,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:38,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:39,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:40,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:00:40,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:41,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:42,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:43,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:44,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:44,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:00:44,631 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:00:44,631 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:00:44,633 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:00:44,673 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:00:45,281 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:00:45,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:46,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:00:46,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:46,675 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:00:47,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:00:47,422 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:00:47,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:48,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:49,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:50,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:51,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:00:51,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:52,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:53,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:54,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:55,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:56,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:00:56,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:57,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:58,300 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:00:58,302 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:00:58,302 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:00:58,304 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:00:58,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:00:58,683 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:00:58,684 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:00:59,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:00,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:01,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:02,342 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:01:02,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:01:02,421 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:01:02,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:03,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:04,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:05,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:06,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:07,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:01:07,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:08,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:09,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:10,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:11,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:12,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:01:12,081 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:01:12,082 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:01:12,084 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:01:12,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:12,694 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:01:12,694 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:01:13,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:01:13,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:14,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:15,285 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:01:15,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:16,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:17,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:01:17,422 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:01:17,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:18,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:01:18,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:19,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:20,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:21,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:22,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:23,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:01:23,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:24,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:25,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:25,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:01:25,838 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:01:25,838 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:01:25,840 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:01:26,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:26,704 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:01:26,705 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:01:27,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:28,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:28,878 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:01:29,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:30,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:31,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:32,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:01:32,422 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:01:32,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:33,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:34,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:01:34,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:35,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:36,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:37,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:38,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:39,506 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:01:39,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:01:39,519 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:01:39,520 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:01:39,521 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:01:39,714 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:01:39,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:40,715 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:01:40,722 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:41,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:42,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:43,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:44,559 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:01:44,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:45,289 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:01:45,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:46,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:47,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:01:47,422 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:01:47,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:48,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:49,570 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:01:49,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:50,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:51,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:52,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:53,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:01:53,443 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:01:53,444 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:01:53,445 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:01:53,724 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:01:53,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:54,725 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:01:54,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:55,483 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:01:55,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:56,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:57,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:58,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:01:59,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:00,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:02:00,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:01,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:02,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:02:02,422 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:02:02,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:03,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:04,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:05,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:02:05,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:06,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:07,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:02:07,182 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:02:07,183 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:02:07,184 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:02:07,734 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:02:07,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:08,735 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:02:08,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:09,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:10,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:11,222 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:02:11,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:12,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:13,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:14,851 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:15,292 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:02:15,851 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:16,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:02:16,851 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:17,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:02:17,422 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:02:17,851 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:18,851 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:19,851 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:20,801 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:02:20,803 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:02:20,804 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:02:20,805 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:02:21,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:21,744 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:02:21,843 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:02:22,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:22,745 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:02:23,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:24,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:25,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:26,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:26,843 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:02:27,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:28,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:29,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:30,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:31,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:31,844 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:02:32,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:32,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:02:32,423 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:02:33,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:34,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:34,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:02:34,509 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:02:34,510 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:02:34,511 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:02:34,752 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:02:35,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:36,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:36,754 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:02:37,117 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:37,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:02:38,233 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:39,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:40,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:41,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:42,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:42,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:02:43,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:44,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:45,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:45,296 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:02:46,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:47,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:47,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:02:47,423 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:02:48,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:48,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:02:48,262 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:02:48,263 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:02:48,263 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:02:48,265 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:02:48,763 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:02:48,763 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:02:49,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:50,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:51,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:52,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:53,224 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:53,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:02:54,225 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:55,225 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:56,225 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:57,225 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:58,225 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:02:58,303 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:02:59,225 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:00,225 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:01,225 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:01,996 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:03:01,997 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:03:01,998 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:03:01,999 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:03:02,226 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:02,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:03:02,423 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:03:02,773 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:03:02,774 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:03:03,226 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:03,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:03:04,309 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:05,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:06,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:07,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:08,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:08,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:03:09,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:10,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:11,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:12,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:13,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:13,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:03:14,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:15,300 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:03:15,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:15,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:03:15,709 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:03:15,710 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:03:15,711 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:03:15,784 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:03:16,303 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:16,784 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:03:17,303 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:17,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:03:17,423 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:03:18,303 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:18,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:03:19,434 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:20,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:21,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:22,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:23,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:23,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:03:24,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:25,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:26,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:27,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:28,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:28,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:03:29,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:03:29,342 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:03:29,342 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:03:29,344 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:03:29,569 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:29,796 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:03:30,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:30,797 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:03:31,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:32,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:03:32,423 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:03:32,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:33,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:34,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:03:34,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:35,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:36,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:37,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:38,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:39,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:03:39,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:40,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:41,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:42,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:43,120 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:03:43,122 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:03:43,122 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:03:43,124 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:03:43,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:43,806 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:03:44,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:44,807 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:03:45,162 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:03:45,302 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:03:45,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:46,784 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:47,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:03:47,423 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:03:47,775 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:48,775 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:49,775 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:50,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:03:50,775 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:51,776 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:52,776 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:53,776 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:54,776 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:55,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:03:55,776 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:56,775 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:03:56,776 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:03:56,777 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:03:56,778 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:03:56,816 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:03:57,004 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:57,785 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:58,785 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:03:58,818 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:03:59,785 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:00,785 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:00,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:04:01,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:02,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:04:02,423 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:04:02,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:03,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:04,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:05,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:06,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:04:06,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:07,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:08,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:09,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:10,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:04:10,547 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:04:10,547 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:04:10,549 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:04:10,826 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:04:10,827 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:04:10,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:11,587 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:04:11,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:12,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:13,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:14,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:15,306 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:04:15,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:16,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:17,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:04:17,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:04:17,423 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:04:17,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:18,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:19,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:20,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:21,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:22,574 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:04:22,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:23,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:24,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:04:24,271 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:04:24,271 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:04:24,273 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:04:24,837 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:04:24,838 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:04:24,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:25,918 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:26,918 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:28,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:28,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:04:28,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:29,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:30,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:31,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:32,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:04:32,424 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:04:32,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:33,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:04:33,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:34,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:35,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:36,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:37,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:38,058 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:04:38,060 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:04:38,060 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:04:38,062 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:04:38,846 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:04:38,847 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:04:38,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:39,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:04:39,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:40,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:41,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:43,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:44,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:44,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:04:45,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:45,309 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:04:46,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:47,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:47,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:04:47,424 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:04:48,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:49,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:49,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:04:50,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:51,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:51,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:04:51,573 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:04:51,573 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:04:51,575 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:04:51,855 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:04:52,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:52,860 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:04:53,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:54,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:54,613 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:04:55,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:56,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:57,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:58,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:59,097 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:04:59,614 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:05:00,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:01,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:02,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:02,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:05:02,424 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:05:03,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:04,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:05,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:05,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:05:05,359 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:05:05,359 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:05:05,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:05:05,361 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:05:05,869 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:05:06,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:07,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:07,870 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:05:08,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:09,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:10,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:10,400 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:05:11,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:12,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:13,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:14,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:15,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:15,313 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:05:16,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:16,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:05:17,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:17,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:05:17,424 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:05:18,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:19,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:19,214 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:05:19,215 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:05:19,216 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:05:19,217 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:05:19,879 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:05:20,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:21,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:21,881 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:05:22,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:22,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:05:23,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:24,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:25,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:26,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:27,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:27,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:05:28,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:29,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:30,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:31,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:32,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:32,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:05:32,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:05:32,424 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:05:33,071 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:05:33,073 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:05:33,073 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:05:33,075 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:05:33,303 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:33,889 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:05:34,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:35,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:35,890 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:05:36,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:37,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:38,076 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:05:38,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:39,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:40,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:41,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:42,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:43,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:05:43,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:44,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:45,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:45,317 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:05:46,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:46,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:05:46,944 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:05:46,944 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:05:46,946 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:05:47,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:47,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:05:47,424 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:05:47,898 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:05:48,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:48,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:05:49,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:49,900 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:05:50,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:51,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:52,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:53,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:53,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:05:54,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:55,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:56,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:57,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:58,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:05:58,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:05:59,260 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:00,260 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:00,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:06:00,649 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:06:00,649 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:06:00,651 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:06:00,907 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:06:01,263 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:01,908 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:06:02,260 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:02,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:06:02,424 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:06:03,260 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:03,555 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:06:04,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:05,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:06,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:07,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:08,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:08,556 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:06:09,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:10,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:11,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:12,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:13,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:13,557 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:06:14,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:06:14,278 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:06:14,279 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:06:14,281 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:06:14,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:14,917 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:06:15,320 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:06:15,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:15,919 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:06:16,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:17,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:17,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:06:17,424 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:06:18,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:18,583 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:06:19,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:20,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:21,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:22,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:23,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:23,584 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:06:24,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:25,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:26,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:27,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:27,924 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:06:27,926 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:06:27,926 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:06:27,928 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:06:28,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:28,928 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:06:28,966 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:06:29,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:29,930 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:06:30,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:31,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:32,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:06:32,425 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:06:32,465 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:33,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:34,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:34,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:06:35,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:36,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:37,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:38,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:39,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:39,525 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:06:40,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:41,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:41,611 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:06:41,612 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:06:41,613 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:06:41,614 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:06:41,938 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:06:42,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:43,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:43,940 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:06:44,445 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:44,653 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:06:45,325 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:06:45,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:46,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:47,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:06:47,425 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:06:47,465 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:48,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:49,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:50,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:50,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:06:51,445 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:52,445 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:53,445 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:54,445 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:55,378 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:06:55,380 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:06:55,381 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:06:55,382 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:06:55,608 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:55,948 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:06:56,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:06:56,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:57,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:57,950 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:06:58,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:06:59,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:00,540 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:01,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:07:01,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:02,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:07:02,425 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:07:02,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:03,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:04,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:05,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:06,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:06,587 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:07:07,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:08,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:09,076 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:07:09,078 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:07:09,078 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:07:09,080 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:07:09,540 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:09,959 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:07:10,540 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:11,540 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:11,961 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:07:12,118 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:07:12,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:13,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:14,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:15,328 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:07:15,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:16,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:17,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:07:17,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:07:17,425 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:07:17,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:18,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:19,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:20,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:21,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:22,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:22,588 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:07:22,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:07:22,703 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:07:22,703 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:07:22,705 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:07:22,970 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:07:23,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:23,971 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:07:24,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:25,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:26,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:27,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:27,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:07:28,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:29,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:30,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:31,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:32,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:07:32,425 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:07:32,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:33,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:07:33,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:34,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:35,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:36,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:07:36,212 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:07:36,212 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:07:36,214 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:07:36,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:36,980 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:07:37,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:37,981 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:07:38,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:39,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:07:39,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:40,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:41,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:42,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:43,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:44,253 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:07:44,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:45,331 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:07:45,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:46,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:47,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:07:47,425 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:07:47,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:48,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:49,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:49,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:07:49,718 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:07:49,720 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:07:49,720 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:07:49,721 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:07:49,989 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:07:50,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:51,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:51,991 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:07:52,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:53,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:54,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:54,760 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:07:55,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:56,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:57,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:58,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:59,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:07:59,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:08:00,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:01,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:02,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:08:02,426 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:08:02,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:03,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:08:03,314 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:08:03,315 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:08:03,316 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:08:03,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:03,999 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:08:04,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:05,354 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:08:05,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:06,001 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:08:06,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:07,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:08,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:09,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:10,355 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:08:10,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:11,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:12,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:13,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:14,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:15,334 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:08:15,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:16,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:08:16,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:16,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:08:16,883 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:08:16,883 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:08:16,885 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:08:17,008 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:08:17,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:08:17,426 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:08:17,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:18,010 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:08:18,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:19,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:20,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:21,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:08:21,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:22,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:23,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:24,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:25,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:26,510 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:08:26,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:27,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:28,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:29,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:30,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:08:30,535 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:08:30,535 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:08:30,537 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:08:30,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:31,019 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:08:31,575 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:08:31,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:32,020 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:08:32,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:08:32,426 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:08:32,865 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:33,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:34,862 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:35,862 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:36,577 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:08:36,862 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:37,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:38,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:39,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:40,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:41,578 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:08:41,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:42,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:43,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:44,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:08:44,169 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:08:44,170 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:08:44,171 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:08:44,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:45,029 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:08:45,338 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:08:45,864 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:46,030 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:08:46,864 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:47,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:08:47,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:08:47,425 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:08:47,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:48,876 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:49,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:50,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:51,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:52,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:08:52,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:53,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:54,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:55,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:56,878 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:57,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:08:57,760 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:08:57,762 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:08:57,762 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:08:57,763 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:08:58,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:58,040 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:08:58,898 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:08:59,899 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:00,042 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:09:00,899 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:01,899 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:02,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:09:02,426 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:09:02,922 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:03,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:09:03,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:04,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:05,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:06,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:07,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:08,506 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:09:08,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:09,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:10,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:11,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:09:11,424 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:09:11,425 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:09:11,426 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:09:11,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:12,050 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:09:12,914 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:13,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:14,052 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:09:14,464 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:09:14,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:15,341 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:09:15,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:16,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:17,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:09:17,426 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:09:17,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:18,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:19,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:09:19,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:20,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:21,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:22,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:23,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:24,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:09:24,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:24,955 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:09:24,956 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:09:24,957 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:09:24,959 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:09:25,060 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:09:25,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:26,061 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:09:26,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:27,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:28,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:29,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:29,997 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:09:30,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:31,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:32,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:09:32,426 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:09:32,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:33,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:34,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:35,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:09:35,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:36,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:37,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:38,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:09:38,455 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:09:38,455 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:09:38,457 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:09:38,918 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:39,069 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:09:39,918 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:40,070 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:09:40,918 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:41,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:09:42,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:43,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:44,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:45,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:45,346 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:09:46,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:47,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:47,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:09:47,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:09:47,427 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:09:48,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:49,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:50,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:51,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:51,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:09:51,735 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:09:51,735 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:09:51,737 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:09:52,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:52,078 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:09:52,775 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:09:53,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:54,045 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:54,079 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:09:55,044 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:56,044 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:57,044 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:57,775 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:09:58,044 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:09:59,045 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:00,045 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:01,045 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:02,045 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:02,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:10:02,427 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:10:03,045 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:03,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:10:04,045 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:05,045 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:05,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:10:05,199 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:10:05,199 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:10:05,201 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:10:06,045 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:06,087 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:10:07,046 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:08,046 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:08,089 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:10:09,046 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:09,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:10:10,046 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:11,046 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:12,046 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:13,046 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:14,046 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:14,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:10:15,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:15,349 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:10:16,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:17,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:17,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:10:17,427 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:10:18,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:18,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:10:18,626 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:10:18,626 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:10:18,628 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:10:19,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:19,096 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:10:19,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:10:20,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:20,098 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:10:21,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:22,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:23,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:24,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:24,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:10:25,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:26,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:27,049 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:28,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:29,049 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:29,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:10:30,049 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:31,049 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:32,049 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:32,096 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:10:32,097 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:10:32,098 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:10:32,100 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:10:32,106 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:10:32,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:10:32,427 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:10:33,051 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:34,051 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:34,108 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:10:35,051 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:35,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:10:36,051 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:37,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:38,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:39,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:40,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:40,592 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:10:41,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:42,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:43,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:44,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:45,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:45,353 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:10:45,359 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:10:45,360 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:10:45,361 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:10:45,362 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:10:46,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:46,116 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:10:46,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:10:47,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:47,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:10:47,428 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:10:48,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:48,118 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:10:49,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:50,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:51,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:51,570 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:10:52,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:53,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:54,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:55,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:56,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:56,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:10:57,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:58,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:58,874 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:10:58,875 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:10:58,876 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:10:58,877 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:10:59,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:10:59,126 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:11:00,120 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:00,127 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:11:01,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:01,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:11:02,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:02,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:11:02,428 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:11:03,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:04,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:05,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:06,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:07,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:07,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:11:08,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:09,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:10,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:11,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:12,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:12,418 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:11:12,420 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:11:12,420 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:11:12,422 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:11:13,137 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:11:13,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:13,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:11:14,138 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:11:14,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:15,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:15,355 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:11:16,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:17,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:17,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:11:17,532 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:11:18,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:18,683 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:11:19,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:20,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:21,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:22,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:23,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:23,684 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:11:24,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:25,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:25,801 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:11:25,803 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:11:25,803 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:11:25,805 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:11:26,148 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:11:26,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:27,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:28,150 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:11:28,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:28,844 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:11:29,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:30,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:31,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:32,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:32,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:11:32,518 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:11:33,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:34,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:34,613 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:11:35,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:36,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:37,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:38,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:39,148 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:11:39,150 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:11:39,150 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:11:39,152 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:11:39,158 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:11:39,391 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:40,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:11:40,390 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:41,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:42,161 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:11:42,458 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:43,458 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:44,458 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:45,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:11:45,359 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:11:45,458 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:46,458 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:47,458 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:47,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:11:47,519 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:11:48,459 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:49,459 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:50,459 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:50,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:11:51,459 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:52,459 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:52,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:11:52,606 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:11:52,606 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:11:52,608 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:11:53,170 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:11:53,459 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:54,170 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:11:54,459 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:55,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:55,646 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:11:56,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:57,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:58,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:11:59,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:00,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:00,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:12:01,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:02,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:02,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:12:02,519 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:12:03,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:04,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:05,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:06,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:12:06,055 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:12:06,055 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:12:06,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:12:06,057 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:12:06,180 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:12:06,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:07,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:08,181 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:12:08,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:09,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:10,462 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:11,095 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:12:11,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:12,462 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:13,462 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:14,462 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:15,364 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:12:15,462 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:16,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:12:16,462 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:17,462 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:17,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:12:17,519 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:12:18,462 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:19,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:12:19,444 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:12:19,445 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:12:19,446 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:12:19,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:20,191 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:12:20,576 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:21,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:12:21,577 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:22,193 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:12:22,577 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:23,577 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:24,577 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:25,577 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:26,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:12:26,577 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:27,577 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:28,578 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:29,578 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:30,578 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:31,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:12:31,578 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:32,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:12:32,519 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:12:32,578 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:32,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:12:32,788 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:12:32,789 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:12:32,790 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:12:33,202 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:12:33,578 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:34,203 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:12:34,578 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:35,578 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:36,579 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:36,828 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:12:37,579 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:38,579 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:39,579 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:40,579 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:41,579 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:41,829 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:12:42,579 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:43,579 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:44,580 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:45,367 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:12:45,580 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:45,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:12:45,999 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:12:45,999 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:12:46,001 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:12:46,212 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:12:46,580 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:47,039 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:12:47,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:12:47,519 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:12:47,580 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:48,215 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:12:48,581 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:49,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:50,671 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:51,671 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:52,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:12:52,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:53,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:54,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:55,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:56,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:57,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:12:57,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:58,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:12:59,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:12:59,289 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:12:59,289 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:12:59,291 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:12:59,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:00,224 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:13:00,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:01,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:02,226 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:13:02,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:13:02,519 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:13:02,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:02,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:13:03,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:04,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:05,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:06,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:07,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:07,680 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:13:08,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:09,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:10,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:11,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:12,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:13:12,530 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:13:12,530 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:13:12,532 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:13:12,762 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:13,234 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:13:13,570 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:13:13,759 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:14,235 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:13:14,843 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:15,369 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:13:15,842 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:16,842 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:17,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:13:17,631 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:13:17,842 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:18,716 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:13:18,843 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:19,843 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:20,843 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:21,843 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:22,843 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:23,716 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:13:23,843 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:24,843 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:25,843 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:25,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:13:25,972 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:13:25,972 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:13:25,974 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:13:26,245 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:13:26,844 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:27,844 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:28,246 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:13:28,844 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:29,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:13:29,844 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:30,844 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:31,844 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:32,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:13:32,631 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:13:32,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:33,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:34,796 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:13:34,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:35,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:36,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:37,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:38,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:39,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:13:39,396 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:13:39,396 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:13:39,398 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:13:39,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:40,255 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:13:40,436 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:13:40,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:41,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:42,257 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:13:42,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:43,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:44,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:45,373 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:13:45,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:46,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:13:46,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:47,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:13:47,631 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:13:47,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:48,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:49,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:50,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:51,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:13:51,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:52,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:13:52,637 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:13:52,638 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:13:52,639 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:13:52,869 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:53,266 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:13:53,866 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:54,267 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:13:54,945 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:55,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:56,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:57,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:13:57,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:58,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:13:59,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:00,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:01,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:02,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:14:02,631 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:14:02,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:14:02,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:03,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:04,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:05,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:06,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:14:06,024 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:14:06,025 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:14:06,026 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:14:06,277 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:14:06,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:07,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:08,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:14:08,278 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:14:08,937 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:09,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:10,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:11,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:12,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:13,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:14:13,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:14,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:15,377 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:14:15,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:16,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:17,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:14:17,631 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:14:17,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:18,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:14:18,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:19,463 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:14:19,464 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:14:19,465 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:14:19,466 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:14:19,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:20,288 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:14:20,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:21,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:22,290 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:14:22,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:23,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:24,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:14:24,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:25,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:26,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:27,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:28,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:29,506 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:14:29,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:30,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:31,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:32,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:14:32,631 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:14:32,726 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:14:32,791 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:14:32,792 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:14:32,793 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:14:32,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:33,299 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:14:33,963 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:34,301 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:14:34,795 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:14:35,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:36,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:37,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:38,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:39,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:39,795 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:14:40,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:41,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:42,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:43,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:44,049 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:44,796 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:14:45,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:45,382 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:14:46,049 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:46,160 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:14:46,161 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:14:46,162 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:14:46,163 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:14:46,310 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:14:47,049 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:47,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:14:47,631 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:14:48,049 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:48,312 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:14:49,049 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:49,799 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:14:50,049 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:51,049 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:52,049 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:53,050 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:54,050 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:54,799 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:14:55,050 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:56,050 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:57,050 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:58,050 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:59,050 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:14:59,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:14:59,538 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:14:59,538 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:14:59,540 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:15:00,050 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:00,321 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:15:00,579 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:15:01,051 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:02,051 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:02,323 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:15:02,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:15:02,631 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:15:03,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:04,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:05,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:05,800 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:15:06,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:07,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:08,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:09,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:10,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:10,801 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:15:11,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:12,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:12,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:15:12,847 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:15:12,848 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:15:12,850 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:15:13,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:13,332 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:15:14,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:14,333 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:15:15,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:15,383 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:15:16,165 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:16,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:15:17,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:17,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:15:17,631 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:15:18,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:19,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:20,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:21,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:21,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:15:22,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:23,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:24,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:25,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:26,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:26,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:15:26,342 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:15:26,343 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:15:26,344 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:15:27,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:27,344 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:15:27,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:15:28,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:28,346 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:15:29,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:30,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:31,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:32,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:32,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:15:32,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:15:32,632 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:15:33,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:34,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:35,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:36,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:37,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:37,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:15:38,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:39,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:39,817 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:15:39,818 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:15:39,819 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:15:39,821 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:15:40,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:40,356 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:15:40,357 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:15:41,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:42,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:42,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:15:43,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:44,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:45,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:45,387 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:15:46,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:47,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:47,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:15:47,632 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:15:48,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:48,786 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:15:49,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:50,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:51,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:52,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:53,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:53,235 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:15:53,237 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:15:53,237 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:15:53,239 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:15:53,366 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:15:54,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:54,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:15:54,367 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:15:55,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:56,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:57,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:58,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:59,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:15:59,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:16:00,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:01,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:02,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:02,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:16:02,632 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:16:03,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:04,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:04,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:16:05,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:06,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:06,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:16:06,514 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:16:06,514 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:16:06,516 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:16:07,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:07,377 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:16:08,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:08,378 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:16:09,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:10,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:10,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:16:11,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:12,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:13,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:14,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:15,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:15,390 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:16:16,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:16,393 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:16:17,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:17,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:16:17,633 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:16:18,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:19,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:19,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:16:19,739 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:16:19,740 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:16:19,741 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:16:20,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:20,387 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:16:20,388 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:16:21,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:21,779 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:16:22,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:23,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:24,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:25,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:26,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:26,780 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:16:27,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:28,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:29,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:30,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:31,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:31,781 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:16:32,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:32,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:16:32,632 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:16:32,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:16:32,916 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:16:32,917 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:16:32,919 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:16:33,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:33,398 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:16:34,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:34,399 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:16:35,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:36,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:36,920 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:16:37,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:38,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:39,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:40,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:41,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:41,921 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:16:42,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:43,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:44,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:45,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:45,394 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:16:45,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:16:46,000 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:16:46,001 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:16:46,002 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:16:46,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:46,408 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:16:46,409 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:16:47,040 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:16:47,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:47,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:16:47,633 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:16:48,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:49,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:50,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:51,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:52,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:52,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:16:53,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:54,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:55,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:56,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:57,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:57,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:16:58,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:59,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:16:59,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:16:59,300 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:16:59,301 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:16:59,302 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:16:59,418 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:17:00,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:00,419 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:17:01,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:02,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:02,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:17:02,633 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:17:03,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:03,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:17:04,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:05,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:06,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:07,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:08,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:08,707 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:17:09,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:10,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:11,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:12,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:12,618 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:17:12,619 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:17:12,620 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:17:12,621 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:17:13,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:13,430 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:17:14,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:14,432 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:17:14,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:17:15,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:15,398 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:17:16,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:17,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:17,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:17:17,674 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:17:18,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:19,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:19,813 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:17:20,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:21,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:22,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:23,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:24,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:24,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:17:25,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:25,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:17:25,842 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:17:25,842 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:17:25,844 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:17:26,339 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:26,443 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:17:26,443 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:17:27,338 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:28,338 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:29,338 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:29,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:17:30,339 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:31,339 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:32,339 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:32,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:17:32,661 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:17:33,339 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:34,339 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:35,339 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:35,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:17:36,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:37,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:38,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:39,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:17:39,115 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:17:39,116 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:17:39,117 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:17:39,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:39,453 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:17:40,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:40,454 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:17:41,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:17:41,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:42,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:43,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:44,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:45,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:45,401 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:17:46,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:46,403 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:17:47,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:47,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:17:47,661 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:17:48,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:49,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:50,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:51,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:51,782 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:17:52,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:17:52,278 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:17:52,279 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:17:52,280 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:17:52,464 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:17:52,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:53,411 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:54,411 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:54,465 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:17:55,411 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:56,411 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:57,319 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:17:57,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:58,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:17:59,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:00,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:01,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:02,320 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:18:02,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:02,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:18:02,662 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:18:03,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:04,413 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:05,413 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:05,483 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:18:05,484 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:18:05,485 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:18:05,486 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:18:06,413 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:06,478 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:18:06,479 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:18:07,413 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:07,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:18:08,413 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:09,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:10,413 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:11,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:12,413 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:12,525 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:18:13,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:14,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:15,405 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:18:15,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:16,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:17,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:17,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:18:17,662 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:18:17,800 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:18:18,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:18,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:18:18,487 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:18:18,487 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:18:18,489 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:18:18,490 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:18:19,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:20,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:20,492 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:18:21,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:22,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:23,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:23,527 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:18:24,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:25,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:26,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:27,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:28,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:28,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:18:29,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:30,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:31,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:31,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:18:31,608 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:18:31,608 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:18:31,610 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:18:32,418 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:32,500 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:18:32,501 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:18:32,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:18:32,662 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:18:33,418 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:33,794 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:18:34,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:35,418 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:36,418 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:37,418 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:38,418 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:38,795 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:18:39,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:40,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:41,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:42,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:43,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:43,796 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:18:44,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:44,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:18:44,902 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:18:44,902 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:18:44,904 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:18:45,408 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:18:45,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:45,509 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:18:46,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:46,510 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:18:47,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:47,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:18:47,662 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:18:48,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:48,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:18:49,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:50,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:51,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:52,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:53,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:53,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:18:54,478 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:55,478 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:56,478 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:57,478 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:58,095 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:18:58,097 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:18:58,097 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:18:58,099 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:18:58,478 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:18:58,520 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:18:58,520 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:18:59,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:18:59,478 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:00,478 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:01,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:02,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:02,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:19:02,662 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:19:03,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:04,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:04,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:19:05,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:06,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:07,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:08,480 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:09,480 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:09,799 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:19:10,480 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:11,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:19:11,417 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:19:11,418 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:19:11,419 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:19:11,535 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:19:11,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:12,536 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:19:12,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:13,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:14,726 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:15,412 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:19:15,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:19:15,726 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:16,726 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:17,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:19:17,662 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:19:17,726 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:18,726 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:19,726 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:20,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:20,792 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:19:21,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:22,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:23,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:24,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:24,949 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:19:24,951 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:19:24,951 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:19:24,953 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:19:25,545 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:19:25,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:25,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:19:26,547 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:19:26,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:27,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:28,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:29,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:30,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:30,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:19:31,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:32,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:19:32,662 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:19:32,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:33,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:34,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:35,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:36,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:36,812 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:19:37,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:38,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:19:38,430 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:19:38,431 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:19:38,433 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:19:38,555 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:19:38,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:39,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:40,557 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:19:40,747 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:41,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:42,470 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:19:42,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:43,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:44,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:45,415 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:19:45,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:46,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:47,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:19:47,663 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:19:47,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:47,785 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:19:48,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:49,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:50,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:51,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:19:51,729 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:19:51,730 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:19:51,731 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:19:51,966 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:52,566 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:19:52,567 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:19:52,965 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:53,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:19:54,049 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:55,046 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:56,046 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:57,046 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:58,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:19:58,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:19:59,046 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:00,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:01,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:02,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:02,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:20:02,663 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:20:03,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:03,825 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:20:04,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:05,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:05,134 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:20:05,136 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:20:05,137 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:20:05,138 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:20:05,576 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:20:06,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:06,577 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:20:07,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:08,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:09,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:09,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:20:10,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:11,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:12,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:13,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:14,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:14,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:20:15,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:15,420 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:20:16,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:17,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:17,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:20:17,663 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:20:18,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:18,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:20:18,456 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:20:18,456 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:20:18,458 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:20:18,587 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:20:19,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:19,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:20:20,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:20,589 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:20:21,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:22,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:23,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:24,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:24,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:20:25,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:26,172 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:27,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:28,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:29,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:29,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:20:30,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:31,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:31,834 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:20:31,836 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:20:31,836 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:20:31,837 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:20:32,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:32,598 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:20:32,598 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:20:32,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:20:32,663 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:20:33,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:34,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:34,820 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:20:35,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:36,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:37,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:38,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:39,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:39,821 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:20:40,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:41,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:42,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:43,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:44,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:44,822 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:20:45,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:45,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:20:45,259 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:20:45,259 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:20:45,261 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:20:45,422 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:20:45,607 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:20:46,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:46,608 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:20:47,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:47,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:20:47,663 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:20:48,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:49,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:50,290 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:50,801 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:20:51,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:52,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:53,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:54,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:55,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:55,802 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:20:56,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:57,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:58,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:20:58,575 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:20:58,576 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:20:58,577 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:20:58,578 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:20:58,616 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:20:59,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:00,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:00,618 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:21:01,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:01,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:21:02,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:02,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:21:02,663 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:21:03,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:04,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:05,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:06,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:06,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:21:07,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:08,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:09,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:10,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:11,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:11,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:21:12,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:21:12,012 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:21:12,013 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:21:12,014 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:21:12,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:12,626 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:21:12,627 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:21:13,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:14,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:15,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:15,425 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:21:16,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:17,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:17,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:21:17,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:21:17,663 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:21:18,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:19,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:20,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:21,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:22,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:22,782 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:21:23,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:24,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:25,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:25,409 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:21:25,410 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:21:25,411 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:21:25,412 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:21:25,636 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:21:26,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:26,637 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:21:27,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:28,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:28,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:21:29,323 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:30,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:31,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:32,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:32,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:21:32,663 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:21:33,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:33,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:21:34,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:35,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:36,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:37,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:38,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:38,719 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:21:38,721 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:21:38,721 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:21:38,723 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:21:38,760 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:21:39,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:39,646 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:21:40,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:40,647 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:21:41,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:42,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:43,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:43,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:21:44,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:45,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:45,429 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:21:46,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:47,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:47,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:21:47,664 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:21:48,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:48,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:21:49,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:50,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:51,418 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:52,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:21:52,104 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:21:52,104 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:21:52,106 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:21:52,418 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:52,659 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:21:53,418 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:53,659 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:21:54,144 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:21:54,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:55,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:56,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:57,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:58,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:21:59,145 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:21:59,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:00,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:01,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:02,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:02,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:22:02,664 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:22:03,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:04,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:04,781 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:22:05,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:05,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:22:05,524 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:22:05,525 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:22:05,527 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:22:05,672 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:22:06,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:07,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:07,674 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:22:08,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:09,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:10,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:10,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:22:11,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:12,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:13,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:14,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:15,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:15,431 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:22:16,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:16,433 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:22:17,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:17,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:22:17,664 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:22:18,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:18,904 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:22:18,905 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:22:18,906 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:22:18,908 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:22:19,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:19,683 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:22:20,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:21,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:21,686 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:22:21,946 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:22:22,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:23,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:24,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:25,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:26,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:26,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:22:27,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:28,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:29,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:30,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:31,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:31,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:22:32,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:32,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:22:32,664 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:22:33,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:34,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:35,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:36,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:37,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:37,825 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:22:38,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:39,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:40,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:41,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:42,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:42,825 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:22:43,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:44,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:45,435 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:22:47,159 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:47,704 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:22:48,437 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:22:52,924 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:22:53,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:22:53,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:54,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:22:54,109 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:22:55,380 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:22:59,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:23:01,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:23:02,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:03,095 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:04,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:23:09,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:23:09,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:23:09,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:10,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:23:10,377 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:23:10,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:11,762 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:14,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:23:15,438 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:23:18,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:23:19,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:20,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:23:20,727 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:25,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:23:27,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:23:28,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:29,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:23:29,620 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:23:29,907 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:30,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:23:35,694 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:23:37,825 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:23:40,695 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:23:42,050 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:42,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:42,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:23:42,486 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:23:43,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:43,300 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:23:44,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:45,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:45,302 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:23:45,441 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:23:46,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:46,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:23:47,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:48,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:49,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:50,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:51,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:51,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:23:52,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:53,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:54,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:55,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:56,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:56,445 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:23:57,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:57,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:23:57,487 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:23:58,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:23:59,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:00,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:01,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:01,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:24:02,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:03,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:04,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:05,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:06,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:06,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:24:07,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:08,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:09,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:10,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:11,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:11,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:24:12,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:12,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:24:12,486 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:24:13,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:14,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:15,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:15,445 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:24:16,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:17,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:17,447 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:24:18,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:19,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:20,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:21,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:22,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:22,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:24:23,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:24,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:25,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:26,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:27,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:27,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:24:27,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:24:27,487 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:24:28,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:29,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:30,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:31,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:32,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:32,610 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:24:33,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:34,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:35,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:36,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:37,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:37,611 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:24:38,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:39,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:24:42,612 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:24:45,448 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:24:48,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:24:53,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:24:58,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:25:01,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:25:01,211 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:01,211 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:25:01,212 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:25:02,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:03,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:04,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:04,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:25:05,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:06,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:07,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:08,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:09,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:09,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:25:10,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:11,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:12,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:13,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:14,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:14,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:25:15,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:15,452 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:25:16,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:25:16,167 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:25:16,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:17,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:18,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:19,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:19,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:25:20,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:21,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:22,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:23,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:24,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:24,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:25:25,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:26,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:27,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:28,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:29,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:29,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:25:30,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:31,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:25:31,167 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:25:31,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:32,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:33,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:34,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:34,324 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:25:35,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:36,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:37,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:38,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:39,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:39,324 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:25:40,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:41,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:42,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:43,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:44,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:44,325 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:25:45,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:45,457 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:25:46,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:25:46,167 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:25:46,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:47,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:48,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:49,184 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:49,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:25:50,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:51,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:52,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:53,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:54,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:54,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:25:55,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:56,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:57,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:58,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:59,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:25:59,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:26:00,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:01,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:26:01,167 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:26:01,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:02,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:03,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:04,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:05,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:05,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:26:06,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:07,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:08,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:09,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:10,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:10,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:26:11,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:12,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:13,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:14,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:15,179 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:15,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:26:15,459 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:26:16,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:26:16,167 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:26:16,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:17,179 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:18,179 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:19,179 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:20,179 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:20,296 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:26:21,180 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:22,179 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:23,179 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:24,180 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:25,180 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:25,297 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:26:26,180 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:27,180 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:28,180 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:29,180 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:30,180 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:30,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:26:31,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:26:31,167 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:26:31,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:32,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:33,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:34,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:35,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:36,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:36,263 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:26:37,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:38,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:39,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:40,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:41,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:41,264 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:26:42,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:43,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:44,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:45,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:45,462 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:26:46,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:26:46,167 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:26:46,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:46,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:26:47,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:48,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:49,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:50,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:51,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:51,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:26:52,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:53,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:54,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:55,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:56,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:26:56,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:26:57,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:01,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:27:06,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:27:11,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:27:15,465 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:27:16,468 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:27:21,468 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:27:26,469 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:27:31,470 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:27:36,471 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:27:39,804 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:27:39,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:27:39,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:39,846 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:27:40,805 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:41,805 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:42,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:27:42,805 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:43,621 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:27:43,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:44,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:45,469 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:27:45,623 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:27:45,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:46,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:47,471 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:27:47,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:48,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:49,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:50,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:51,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:52,472 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:27:52,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:53,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:54,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:54,809 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:27:54,810 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:27:55,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:55,950 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:27:55,952 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:27:55,953 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:27:55,954 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:27:56,630 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:27:56,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:57,631 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:27:57,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:57,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:27:58,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:27:59,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:00,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:01,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:02,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:02,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:28:03,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:04,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:05,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:06,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:07,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:07,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:28:08,809 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:09,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:28:09,492 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:28:09,493 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:28:09,494 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:28:09,640 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:28:09,809 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:09,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:28:09,810 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:28:10,809 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:11,641 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:28:11,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:12,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:13,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:13,924 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:28:14,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:15,473 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:28:15,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:16,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:17,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:18,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:19,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:28:19,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:20,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:21,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:22,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:23,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:28:23,048 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:28:23,048 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:28:23,049 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:28:23,650 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:28:23,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:24,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:28:24,810 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:28:24,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:24,898 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:28:25,651 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:28:25,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:26,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:27,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:28,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:29,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:29,899 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:28:30,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:31,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:32,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:33,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:34,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:34,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:28:35,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:36,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:28:36,385 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:28:36,386 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:28:36,387 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:28:36,658 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:28:36,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:37,659 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:28:37,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:38,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:39,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:28:39,811 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:28:39,851 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:39,974 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:28:40,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:41,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:42,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:43,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:44,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:44,975 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:28:45,477 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:28:45,820 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:46,820 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:47,820 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:48,820 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:49,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:28:49,752 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:28:49,753 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:28:49,754 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:28:49,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:50,667 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:28:50,793 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:28:50,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:51,668 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:28:51,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:52,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:53,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:54,843 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:28:54,844 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:28:54,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:55,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:55,955 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:28:56,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:57,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:58,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:28:59,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:00,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:00,956 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:29:01,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:02,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:03,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:29:03,225 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:29:03,225 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:29:03,227 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:29:03,677 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:29:03,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:04,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:05,679 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:29:05,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:06,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:29:06,902 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:07,899 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:08,899 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:09,826 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:29:09,826 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:29:09,899 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:10,899 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:11,899 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:11,962 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:29:12,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:13,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:14,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:15,482 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:29:15,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:16,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:29:16,705 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:29:16,706 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:29:16,707 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:29:16,929 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:17,688 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:29:17,688 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:29:17,747 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:29:17,928 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:19,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:20,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:21,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:22,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:22,747 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:29:23,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:24,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:24,826 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:29:24,827 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:29:25,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:26,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:27,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:27,919 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:29:28,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:29,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:30,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:30,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:29:30,240 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:29:30,241 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:29:30,243 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:29:30,698 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:29:31,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:31,700 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:29:32,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:33,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:33,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:29:34,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:35,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:36,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:37,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:38,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:38,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:29:39,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:39,826 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:29:39,827 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:29:40,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:41,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:42,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:43,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:43,646 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:29:43,648 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:29:43,648 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:29:43,649 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:29:43,650 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:29:43,708 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:29:44,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:45,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:45,484 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:29:45,709 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:29:46,038 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:47,036 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:48,036 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:49,036 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:49,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:29:50,036 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:51,036 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:52,036 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:53,037 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:54,037 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:54,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:29:54,826 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:29:54,827 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:29:55,037 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:56,037 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:57,037 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:57,173 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:29:57,174 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:29:57,175 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:29:57,176 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:29:57,718 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:29:58,038 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:59,037 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:29:59,720 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:30:00,038 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:00,215 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:30:01,038 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:02,037 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:03,038 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:04,038 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:05,038 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:05,216 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:30:06,038 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:07,038 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:08,038 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:09,038 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:09,826 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:30:09,827 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:30:10,039 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:10,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:30:10,498 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:30:10,499 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:30:10,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:30:10,501 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:30:10,728 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:30:11,039 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:11,729 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:30:12,039 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:13,039 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:14,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:15,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:15,488 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:30:16,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:16,490 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:30:17,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:18,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:19,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:20,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:21,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:21,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:30:22,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:23,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:23,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:30:23,890 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:30:23,891 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:30:23,893 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:30:24,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:24,739 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:30:24,826 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:30:24,827 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:30:25,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:25,741 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:30:26,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:26,969 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:30:27,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:28,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:29,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:30,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:31,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:31,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:30:32,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:33,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:34,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:35,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:36,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:36,971 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:30:37,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:37,319 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:30:37,320 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:30:37,321 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:30:37,322 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:30:37,749 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:30:38,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:39,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:39,751 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:30:39,827 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:30:39,827 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:30:40,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:41,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:42,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:42,895 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:30:43,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:44,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:45,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:45,492 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:30:46,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:47,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:48,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:48,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:30:49,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:50,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:50,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:30:50,743 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:30:50,743 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:30:50,745 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:30:50,759 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:30:51,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:51,760 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:30:52,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:53,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:53,783 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:30:54,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:54,827 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:30:54,827 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:30:55,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:56,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:57,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:58,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:30:58,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:30:59,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:00,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:01,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:02,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:03,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:03,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:31:04,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:31:04,191 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:31:04,191 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:31:04,193 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:31:04,431 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:04,770 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:31:05,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:05,771 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:31:06,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:07,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:08,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:09,232 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:31:09,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:09,827 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:31:09,827 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:31:10,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:11,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:12,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:13,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:14,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:14,979 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:31:15,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:15,495 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:31:16,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:17,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:17,612 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:31:17,614 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:31:17,614 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:31:17,616 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:31:17,780 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:31:18,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:19,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:19,782 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:31:20,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:20,654 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:31:21,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:22,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:23,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:24,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:24,827 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:31:24,828 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:31:25,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:25,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:31:26,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:27,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:28,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:29,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:30,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:30,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:31:31,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:31:31,171 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:31:31,172 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:31:31,173 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:31:31,401 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:31,791 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:31:32,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:33,483 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:33,792 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:31:34,482 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:35,483 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:36,211 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:31:36,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:37,483 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:38,483 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:39,483 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:39,827 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:31:39,828 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:31:40,483 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:41,483 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:41,901 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:31:42,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:43,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:44,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:44,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:31:44,671 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:31:44,672 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:31:44,673 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:31:44,801 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:31:45,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:45,500 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:31:45,801 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:31:46,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:47,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:47,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:31:48,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:49,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:50,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:51,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:52,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:52,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:31:53,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:54,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:54,827 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:31:54,828 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:31:55,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:56,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:57,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:57,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:31:58,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:31:58,092 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:31:58,092 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:31:58,094 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:31:58,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:58,811 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:31:59,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:31:59,812 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:32:00,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:01,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:02,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:03,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:32:03,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:04,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:05,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:06,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:07,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:08,133 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:32:08,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:09,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:09,828 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:32:09,828 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:32:10,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:11,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:11,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:32:11,632 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:32:11,633 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:32:11,634 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:32:11,821 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:32:12,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:13,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:13,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:32:13,823 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:32:14,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:15,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:15,502 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:32:16,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:17,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:18,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:19,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:19,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:32:20,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:21,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:22,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:23,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:24,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:24,506 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:32:24,828 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:32:24,828 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:32:25,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:32:25,152 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:32:25,153 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:32:25,154 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:32:25,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:25,836 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:32:25,836 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:32:26,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:27,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:28,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:29,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:30,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:32:30,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:31,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:32,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:33,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:34,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:35,157 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:32:35,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:36,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:37,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:38,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:38,671 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:32:38,672 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:32:38,672 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:32:38,674 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:32:38,844 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:32:39,490 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:39,828 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:32:39,828 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:32:39,845 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:32:40,490 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:40,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:32:41,490 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:42,490 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:43,490 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:44,490 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:45,490 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:45,506 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:32:46,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:46,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:32:47,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:48,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:49,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:50,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:51,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:51,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:32:52,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:32:52,090 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:32:52,091 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:32:52,092 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:32:52,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:52,853 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:32:53,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:53,854 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:32:54,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:54,828 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:32:54,829 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:32:55,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:56,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:56,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:32:57,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:58,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:32:59,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:00,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:01,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:01,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:33:02,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:03,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:04,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:05,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:05,713 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:33:05,714 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:33:05,714 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:33:05,716 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:33:05,863 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:33:06,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:07,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:07,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:33:07,864 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:33:08,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:09,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:09,828 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:33:09,829 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:33:10,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:11,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:12,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:12,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:33:13,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:14,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:15,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:15,510 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:33:16,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:17,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:18,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:18,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:33:19,147 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:33:19,148 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:33:19,149 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:33:19,150 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:33:19,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:19,873 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:33:19,873 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:33:20,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:21,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:22,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:23,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:24,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:33:24,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:24,828 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:33:24,829 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:33:25,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:26,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:27,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:28,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:29,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:29,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:33:30,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:31,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:32,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:32,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:33:32,542 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:33:32,543 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:33:32,544 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:33:32,882 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:33:33,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:33,883 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:33:34,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:35,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:35,583 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:33:36,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:37,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:38,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:39,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:39,828 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:33:39,829 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:33:40,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:40,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:33:41,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:42,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:43,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:44,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:45,513 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:33:45,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:46,008 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:33:46,009 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:33:46,010 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:33:46,010 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:33:46,011 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:33:46,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:46,892 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:33:47,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:47,893 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:33:48,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:49,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:50,683 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:51,050 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:33:51,683 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:52,684 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:53,684 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:54,684 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:54,828 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:33:54,829 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:33:55,684 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:56,684 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:57,008 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:33:57,684 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:58,684 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:59,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:33:59,245 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:33:59,246 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:33:59,247 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:33:59,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:33:59,902 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:33:59,903 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:34:00,684 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:01,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:02,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:34:02,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:03,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:04,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:05,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:06,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:07,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:34:07,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:08,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:09,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:09,829 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:34:09,829 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:34:10,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:11,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:12,435 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:34:12,436 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:34:12,437 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:34:12,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:34:12,439 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:34:12,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:12,911 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:34:13,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:13,912 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:34:14,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:15,518 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:34:15,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:16,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:17,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:34:17,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:18,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:19,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:20,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:21,753 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:22,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:34:22,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:23,753 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:24,753 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:24,829 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:34:24,829 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:34:25,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:34:25,714 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:34:25,715 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:34:25,716 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:34:25,921 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:34:25,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:26,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:27,718 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:34:27,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:27,924 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:34:28,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:29,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:30,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:31,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:32,718 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:34:32,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:33,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:34,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:35,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:36,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:37,719 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:34:37,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:38,800 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:34:38,802 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:34:38,802 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:34:38,804 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:34:38,934 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:34:39,038 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:39,831 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:34:39,832 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:34:39,934 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:34:39,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:40,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:41,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:42,925 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:34:42,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:43,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:44,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:45,521 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:34:45,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:46,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:47,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:48,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:34:48,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:49,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:50,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:51,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:52,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:34:52,035 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:34:52,036 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:34:52,037 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:34:52,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:52,942 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:34:53,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:53,943 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:34:54,075 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:34:54,831 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:34:54,832 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:34:54,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:55,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:56,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:57,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:58,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:34:59,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:34:59,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:00,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:01,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:02,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:03,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:04,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:35:04,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:05,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:35:05,336 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:35:05,337 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:35:05,338 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:35:05,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:05,951 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:35:05,951 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:35:06,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:07,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:09,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:09,831 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:35:09,831 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:35:10,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:10,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:35:11,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:12,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:13,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:14,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:15,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:15,524 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:35:16,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:16,527 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:35:17,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:18,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:18,618 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:35:18,620 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:35:18,620 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:35:18,622 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:35:18,959 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:35:19,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:19,960 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:35:20,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:21,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:21,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:35:22,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:23,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:24,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:24,831 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:35:24,832 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:35:25,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:26,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:26,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:35:27,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:28,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:29,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:30,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:31,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:31,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:35:32,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:32,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:35:32,023 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:35:32,024 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:35:32,025 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:35:32,969 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:35:33,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:33,970 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:35:34,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:35,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:36,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:37,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:37,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:35:38,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:39,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:39,831 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:35:39,832 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:35:40,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:41,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:42,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:42,949 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:35:43,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:44,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:45,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:45,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:35:45,240 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:35:45,241 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:35:45,243 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:35:45,528 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:35:45,978 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:35:45,979 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:35:46,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:47,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:48,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:48,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:35:49,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:50,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:51,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:52,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:53,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:53,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:35:54,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:54,831 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:35:54,832 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:35:55,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:56,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:57,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:58,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:58,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:35:58,461 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:35:58,462 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:35:58,464 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:35:58,987 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:35:59,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:35:59,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:35:59,988 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:36:00,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:01,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:02,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:03,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:04,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:04,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:36:05,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:06,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:07,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:08,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:09,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:09,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:36:09,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:36:09,832 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:36:10,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:11,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:11,757 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:36:11,759 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:36:11,759 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:36:11,761 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:36:11,995 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:36:12,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:13,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:13,997 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:36:14,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:14,799 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:36:15,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:15,532 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:36:16,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:17,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:18,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:19,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:20,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:20,534 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:36:21,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:22,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:23,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:24,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:24,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:36:24,832 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:36:25,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:36:25,054 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:36:25,054 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:36:25,056 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:36:25,291 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:26,005 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:36:26,006 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:36:26,057 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:36:26,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:27,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:28,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:29,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:30,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:31,057 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:36:31,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:32,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:33,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:34,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:35,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:36,058 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:36:36,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:37,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:38,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:38,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:36:38,206 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:36:38,207 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:36:38,209 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:36:39,014 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:36:39,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:39,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:36:39,832 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:36:40,015 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:36:40,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:41,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:41,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:36:42,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:43,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:44,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:45,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:45,535 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:36:46,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:47,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:47,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:36:48,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:49,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:50,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:51,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:51,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:36:51,444 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:36:51,444 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:36:51,446 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:36:52,023 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:36:52,023 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:36:52,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:53,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:53,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:36:54,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:54,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:36:54,833 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:36:55,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:56,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:57,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:58,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:36:58,906 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:36:59,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:00,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:01,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:02,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:03,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:03,906 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:37:04,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:04,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:37:04,722 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:37:04,723 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:37:04,724 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:37:05,032 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:37:05,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:06,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:07,034 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:37:07,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:08,402 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:09,386 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:09,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:37:09,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:37:09,833 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:37:10,386 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:11,386 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:12,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:13,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:14,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:14,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:37:15,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:15,539 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:37:16,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:17,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:17,945 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:37:17,946 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:37:17,947 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:37:17,948 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:37:18,042 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:37:18,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:19,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:19,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:37:20,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:21,045 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:37:21,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:22,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:23,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:24,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:24,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:37:24,833 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:37:25,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:25,966 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:37:26,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:27,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:28,389 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:29,389 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:30,389 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:30,966 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:37:31,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:37:31,210 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:37:31,210 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:37:31,212 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:37:31,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:32,053 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:37:32,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:33,054 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:37:33,527 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:34,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:35,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:36,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:37:36,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:37,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:38,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:39,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:39,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:37:39,833 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:37:40,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:41,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:41,919 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:37:42,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:43,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:44,403 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:37:44,404 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:37:44,405 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:37:44,406 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:37:44,639 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:45,063 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:37:45,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:45,543 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:37:46,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:47,065 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:37:47,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:47,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:37:48,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:49,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:50,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:51,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:52,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:52,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:37:53,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:54,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:54,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:37:54,833 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:37:55,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:56,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:57,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:57,746 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:37:57,747 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:37:57,748 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:37:57,748 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:37:57,749 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:37:58,073 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:37:58,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:37:59,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:00,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:01,075 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:38:01,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:02,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:02,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:38:03,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:04,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:05,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:06,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:07,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:07,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:38:08,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:09,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:09,833 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:38:09,833 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:38:10,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:11,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:38:11,081 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:38:11,082 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:38:11,083 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:38:11,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:12,084 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:38:12,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:13,085 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:38:13,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:38:13,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:14,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:15,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:15,546 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:38:16,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:17,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:18,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:18,548 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:38:19,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:20,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:21,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:22,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:23,544 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:23,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:38:24,372 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:38:24,373 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:38:24,374 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:38:24,375 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:38:24,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:24,833 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:38:24,833 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:38:25,094 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:38:25,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:26,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:27,095 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:38:27,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:28,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:28,955 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:38:29,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:30,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:31,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:32,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:33,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:33,955 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:38:34,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:35,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:36,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:37,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:38:37,595 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:38:37,595 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:38:37,597 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:38:37,830 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:38,104 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:38:38,729 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:39,105 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:38:39,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:38:39,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:39,904 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:38:39,904 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:38:40,729 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:41,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:42,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:43,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:44,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:45,070 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:38:45,549 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:38:45,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:46,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:47,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:48,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:49,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:50,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:38:50,718 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:38:50,719 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:38:50,720 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:38:50,722 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:38:50,953 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:51,114 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:38:51,853 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:52,853 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:53,116 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:38:53,854 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:54,854 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:54,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:38:54,889 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:38:55,854 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:55,953 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:38:56,854 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:57,854 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:58,854 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:38:59,854 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:00,854 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:00,953 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:39:01,854 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:02,855 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:03,774 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:39:03,776 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:39:03,776 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:39:03,778 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:39:04,005 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:04,124 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:39:04,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:05,125 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:39:05,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:06,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:39:06,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:07,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:08,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:09,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:39:09,889 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:39:09,929 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:10,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:11,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:12,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:39:12,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:13,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:14,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:15,553 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:39:15,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:16,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:39:16,890 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:39:16,891 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:39:16,892 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:39:17,120 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:17,134 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:39:17,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:39:18,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:19,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:19,136 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:39:20,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:21,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:22,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:22,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:39:23,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:24,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:24,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:39:24,890 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:39:25,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:26,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:27,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:28,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:28,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:39:29,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:30,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:30,126 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:39:30,127 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:39:30,128 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:39:30,129 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:39:30,144 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:39:31,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:31,145 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:39:32,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:33,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:33,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:39:34,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:35,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:36,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:37,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:38,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:38,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:39:39,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:39,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:39:39,890 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:39:40,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:41,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:42,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:43,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:43,393 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:39:43,395 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:39:43,395 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:39:43,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:39:43,397 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:39:44,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:44,155 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:39:45,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:45,156 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:39:45,557 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:39:46,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:47,093 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:48,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:48,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:39:49,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:50,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:51,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:52,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:53,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:53,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:39:54,078 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:54,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:39:54,890 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:39:55,078 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:56,078 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:56,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:39:56,678 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:39:56,678 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:39:56,680 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:39:57,078 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:57,165 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:39:58,078 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:58,718 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:39:59,078 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:39:59,166 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:40:00,078 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:01,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:02,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:03,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:03,719 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:40:04,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:05,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:06,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:07,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:08,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:08,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:40:09,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:09,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:40:09,890 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:40:10,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:10,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:40:10,156 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:40:10,157 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:40:10,158 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:40:10,174 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:40:11,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:11,175 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:40:12,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:13,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:14,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:14,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:40:15,179 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:15,560 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:40:16,165 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:17,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:18,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:19,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:19,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:40:20,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:21,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:22,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:23,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:23,304 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:40:23,305 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:40:23,306 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:40:23,307 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:40:24,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:24,184 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:40:24,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:40:24,890 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:40:25,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:40:25,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:25,185 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:40:26,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:27,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:28,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:29,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:30,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:40:30,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:31,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:32,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:33,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:34,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:35,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:40:35,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:36,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:36,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:40:36,603 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:40:36,604 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:40:36,605 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:40:37,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:37,196 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:40:38,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:39,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:39,197 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:40:39,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:40:39,890 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:40:40,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:40,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:40:41,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:42,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:43,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:44,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:45,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:45,564 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:40:46,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:46,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:40:47,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:48,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:49,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:49,707 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:40:49,708 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:40:49,709 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:40:49,710 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:40:50,206 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:40:50,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:51,207 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:40:51,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:51,748 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:40:52,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:53,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:54,372 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:54,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:40:54,891 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:40:55,372 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:56,372 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:56,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:40:57,373 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:58,373 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:40:59,373 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:00,373 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:01,373 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:01,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:41:02,373 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:02,918 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:41:02,920 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:41:02,920 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:41:02,922 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:41:03,217 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:41:03,373 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:04,373 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:05,219 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:41:05,373 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:06,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:06,960 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:41:07,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:08,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:09,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:09,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:41:09,891 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:41:10,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:11,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:12,046 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:41:12,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:13,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:14,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:15,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:15,568 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:41:16,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:41:16,157 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:41:16,157 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:41:16,159 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:41:16,227 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:41:16,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:17,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:41:17,228 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:41:17,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:18,463 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:19,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:20,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:21,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:22,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:41:22,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:23,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:24,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:24,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:41:24,891 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:41:25,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:26,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:27,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:28,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:41:28,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:29,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:41:29,347 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:41:29,348 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:41:29,349 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:41:29,582 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:30,237 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:41:30,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:31,238 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:41:31,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:32,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:33,387 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:41:33,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:34,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:35,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:36,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:37,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:38,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:41:38,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:39,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:39,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:41:39,891 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:41:40,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:41,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:42,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:42,555 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:41:42,557 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:41:42,558 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:41:42,559 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:41:43,248 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:41:43,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:43,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:41:44,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:45,250 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:41:45,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:45,571 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:41:46,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:47,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:48,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:49,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:49,574 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:41:50,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:51,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:52,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:53,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:54,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:54,575 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:41:54,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:41:54,891 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:41:55,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:55,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:41:55,737 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:41:55,738 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:41:55,739 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:41:56,259 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:41:56,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:57,260 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:41:57,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:58,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:59,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:41:59,778 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:42:00,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:01,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:02,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:03,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:04,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:04,779 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:42:05,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:06,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:07,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:08,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:08,928 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:42:08,930 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:42:08,930 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:42:08,932 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:42:09,269 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:42:09,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:09,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:42:09,892 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:42:10,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:42:10,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:11,271 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:42:11,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:12,609 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:13,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:14,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:15,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:42:15,576 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:42:15,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:16,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:17,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:18,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:19,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:20,579 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:42:20,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:21,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:22,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:42:22,091 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:42:22,092 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:42:22,093 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:42:22,278 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:42:22,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:23,279 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:42:23,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:24,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:24,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:42:24,892 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:42:25,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:26,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:42:26,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:27,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:28,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:29,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:30,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:31,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:42:31,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:32,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:33,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:34,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:35,318 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:42:35,320 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:42:35,321 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:42:35,322 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:42:35,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:36,289 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:42:36,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:42:36,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:37,290 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:42:37,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:38,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:39,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:39,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:42:39,892 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:42:40,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:41,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:42,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:42:42,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:43,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:44,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:45,579 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:42:45,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:46,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:47,582 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:42:47,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:48,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:42:48,397 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:42:48,397 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:42:48,399 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:42:48,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:49,301 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:42:49,301 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:42:49,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:50,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:51,697 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:52,698 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:53,437 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:42:53,698 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:54,698 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:54,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:42:54,892 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:42:55,698 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:56,698 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:57,698 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:58,698 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:42:59,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:42:59,698 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:00,698 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:01,469 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:43:01,470 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:43:01,471 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:43:01,473 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:43:01,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:02,311 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:43:02,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:03,312 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:43:03,777 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:04,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:43:04,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:05,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:06,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:07,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:08,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:09,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:43:09,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:09,892 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:43:09,892 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:43:10,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:11,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:12,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:13,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:14,646 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:43:14,647 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:43:14,648 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:43:14,648 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:43:14,650 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:43:14,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:15,321 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:43:15,583 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:43:15,782 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:16,782 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:17,322 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:43:17,782 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:18,783 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:19,782 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:20,585 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:43:20,783 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:21,783 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:22,783 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:23,783 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:24,783 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:24,892 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:43:24,892 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:43:25,783 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:25,977 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:43:26,783 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:27,783 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:27,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:43:27,809 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:43:27,809 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:43:27,810 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:43:28,331 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:43:28,784 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:29,332 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:43:29,784 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:30,784 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:31,784 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:31,849 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:43:32,901 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:33,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:34,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:35,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:36,850 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:43:36,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:37,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:38,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:39,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:39,892 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:43:39,893 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:43:40,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:40,963 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:43:40,965 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:43:40,965 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:43:40,967 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:43:41,340 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:43:41,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:42,006 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:43:42,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:43,343 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:43:43,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:44,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:45,585 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:43:45,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:46,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:47,588 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:43:47,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:48,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:49,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:50,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:51,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:52,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:43:52,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:53,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:54,134 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:43:54,136 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:43:54,137 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:43:54,139 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:43:54,354 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:43:54,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:54,892 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:43:54,892 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:43:55,355 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:43:55,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:56,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:57,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:57,977 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:43:58,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:43:59,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:00,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:01,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:02,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:02,978 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:44:03,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:04,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:05,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:06,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:07,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:44:07,430 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:44:07,430 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:44:07,432 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:44:07,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:08,365 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:44:08,470 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:44:08,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:09,366 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:44:09,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:09,892 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:44:09,892 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:44:10,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:11,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:12,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:13,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:14,043 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:44:14,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:15,589 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:44:15,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:16,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:17,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:18,892 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:19,592 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:44:19,892 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:20,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:44:20,629 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:44:20,630 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:44:20,631 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:44:20,892 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:21,375 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:44:21,892 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:22,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:23,377 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:44:23,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:24,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:44:24,892 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:44:24,893 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:44:24,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:25,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:26,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:27,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:28,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:29,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:30,043 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:44:30,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:31,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:32,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:33,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:44:33,808 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:44:33,809 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:44:33,810 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:44:34,036 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:34,386 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:44:34,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:35,387 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:44:35,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:44:35,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:36,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:37,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:38,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:39,893 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:44:39,893 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:44:39,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:40,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:41,057 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:44:41,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:42,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:43,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:44,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:45,594 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:44:45,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:46,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:44:46,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:47,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:44:47,105 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:44:47,106 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:44:47,107 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:44:47,397 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:44:47,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:48,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:49,398 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:44:49,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:50,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:51,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:52,145 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:44:52,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:53,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:54,893 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:44:54,893 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:44:54,943 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:55,944 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:56,944 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:57,944 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:58,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:44:58,944 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:44:59,944 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:00,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:45:00,274 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:45:00,274 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:45:00,276 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:45:00,407 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:45:00,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:01,408 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:45:01,945 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:02,945 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:03,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:45:03,960 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:04,946 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:05,946 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:06,946 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:07,946 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:08,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:45:08,946 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:09,893 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:45:09,894 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:45:09,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:10,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:11,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:12,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:13,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:45:13,556 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:45:13,557 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:45:13,557 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:45:13,558 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:45:13,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:14,417 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:45:14,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:15,419 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:45:15,595 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:45:15,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:16,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:17,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:18,597 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:45:18,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:19,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:20,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:21,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:22,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:23,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:45:23,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:24,893 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:45:24,893 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:45:24,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:25,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:26,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:45:26,769 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:45:26,769 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:45:26,771 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:45:26,997 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:27,428 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:45:27,995 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:28,809 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:45:29,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:29,430 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:45:30,074 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:31,074 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:32,074 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:33,075 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:33,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:45:34,075 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:35,075 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:36,075 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:37,075 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:38,075 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:38,811 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:45:39,075 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:39,893 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:45:39,894 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:45:39,944 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:45:39,979 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:45:39,980 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:45:39,982 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:45:40,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:40,439 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:45:41,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:41,440 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:45:42,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:43,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:43,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:45:44,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:45,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:45,598 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:45:46,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:47,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:48,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:49,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:49,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:45:50,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:51,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:52,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:53,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:53,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:45:53,248 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:45:53,249 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:45:53,250 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:45:53,449 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:45:54,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:54,893 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:45:54,894 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:45:54,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:45:55,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:55,451 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:45:56,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:57,088 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:58,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:59,088 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:45:59,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:46:00,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:01,088 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:02,088 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:03,088 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:04,088 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:04,984 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:46:05,088 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:06,088 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:06,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:46:06,359 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:46:06,360 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:46:06,362 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:46:06,460 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:46:07,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:07,461 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:46:08,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:09,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:10,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:46:10,026 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:46:10,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:10,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:46:11,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:12,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:13,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:14,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:15,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:15,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:46:15,599 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:46:16,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:17,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:18,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:19,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:19,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:46:19,486 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:46:19,487 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:46:19,489 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:46:20,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:20,471 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:46:20,526 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:46:21,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:21,473 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:46:22,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:23,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:24,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:25,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:46:25,011 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:46:25,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:26,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:46:26,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:27,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:28,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:29,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:30,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:31,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:46:31,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:32,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:32,746 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:46:32,748 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:46:32,749 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:46:32,750 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:46:33,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:33,482 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:46:33,482 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:46:34,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:35,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:36,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:36,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:46:37,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:38,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:39,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:40,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:46:40,012 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:46:40,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:41,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:42,129 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:46:42,170 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:43,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:44,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:45,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:45,603 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:46:45,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:46:45,961 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:46:45,962 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:46:45,963 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:46:46,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:46,491 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:46:47,192 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:47,492 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:46:48,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:46:48,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:49,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:50,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:51,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:52,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:53,002 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:46:53,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:54,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:55,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:46:55,012 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:46:55,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:56,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:57,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:58,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:46:58,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:59,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:46:59,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:46:59,338 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:46:59,338 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:46:59,340 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:46:59,501 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:47:00,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:01,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:01,503 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:47:02,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:03,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:03,378 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:47:04,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:05,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:06,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:07,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:08,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:08,379 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:47:09,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:10,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:47:10,012 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:47:10,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:11,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:12,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:12,569 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:47:12,571 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:47:12,571 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:47:12,573 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:47:13,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:13,512 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:47:13,513 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:47:13,611 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:47:14,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:15,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:15,607 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:47:16,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:17,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:18,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:19,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:19,610 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:47:20,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:21,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:22,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:23,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:24,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:24,611 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:47:25,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:47:25,012 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:47:25,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:25,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:47:25,744 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:47:25,745 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:47:25,746 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:47:26,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:26,522 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:47:27,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:27,523 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:47:28,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:29,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:29,785 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:47:30,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:31,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:32,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:33,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:34,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:34,786 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:47:35,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:36,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:37,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:38,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:38,954 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:47:38,956 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:47:38,957 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:47:38,958 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:47:39,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:39,532 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:47:39,996 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:47:40,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:47:40,012 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:47:40,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:41,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:41,534 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:47:42,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:43,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:44,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:45,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:47:45,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:45,610 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:47:46,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:47,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:48,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:49,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:50,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:50,613 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:47:51,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:52,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:52,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:47:52,335 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:47:52,335 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:47:52,337 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:47:52,543 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:47:53,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:53,544 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:47:54,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:55,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:47:55,012 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:47:55,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:56,112 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:47:56,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:57,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:58,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:47:59,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:00,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:01,113 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:48:01,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:02,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:03,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:04,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:05,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:05,681 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:48:05,683 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:48:05,683 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:48:05,685 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:48:06,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:06,554 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:48:06,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:48:07,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:07,555 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:48:08,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:09,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:10,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:48:10,012 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:48:10,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:11,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:12,149 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:48:12,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:13,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:14,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:15,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:15,614 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:48:16,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:17,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:17,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:48:18,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:18,837 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:48:18,839 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:48:18,839 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:48:18,841 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:48:19,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:19,565 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:48:19,566 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:48:20,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:21,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:22,490 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:22,879 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:48:23,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:24,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:25,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:48:25,012 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:48:25,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:26,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:27,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:28,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:48:28,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:29,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:30,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:31,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:31,924 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:48:31,926 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:48:31,926 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:48:31,928 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:48:32,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:32,576 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:48:33,478 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:33,577 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:48:33,966 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:48:34,478 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:35,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:36,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:37,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:38,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:38,967 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:48:39,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:40,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:48:40,013 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:48:40,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:41,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:42,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:43,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:44,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:48:44,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:45,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:48:45,058 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:48:45,058 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:48:45,060 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:48:45,586 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:48:45,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:45,618 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:48:46,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:47,588 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:48:47,592 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:48,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:49,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:49,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:48:50,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:51,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:52,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:53,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:54,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:54,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:48:55,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:48:55,013 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:48:55,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:56,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:57,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:58,458 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:48:58,459 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:48:58,460 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:48:58,462 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:48:58,597 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:48:58,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:48:59,598 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:48:59,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:00,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:49:00,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:01,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:02,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:03,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:04,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:05,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:49:05,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:06,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:07,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:08,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:09,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:10,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:49:10,013 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:49:10,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:11,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:49:11,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:49:11,567 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:49:11,568 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:49:11,569 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:49:11,607 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:49:11,803 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:12,795 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:13,609 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:49:13,869 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:14,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:15,621 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:49:15,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:16,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:49:16,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:17,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:18,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:19,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:20,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:21,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:49:21,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:22,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:23,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:24,748 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:49:24,749 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:49:24,750 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:49:24,752 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:49:24,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:25,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:49:25,013 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:49:25,618 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:49:25,619 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:49:25,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:27,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:27,130 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:49:28,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:29,061 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:30,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:31,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:32,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:32,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:49:33,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:34,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:35,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:36,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:37,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:37,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:49:37,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:49:37,971 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:49:37,972 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:49:37,974 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:49:38,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:38,627 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:49:39,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:39,628 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:49:40,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:49:40,013 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:49:40,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:41,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:42,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:49:42,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:43,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:44,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:45,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:45,625 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:49:46,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:47,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:47,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:49:48,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:49,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:50,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:51,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:49:51,115 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:49:51,116 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:49:51,118 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:49:51,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:51,637 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:49:52,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:53,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:49:53,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:53,639 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:49:54,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:55,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:49:55,013 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:49:55,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:56,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:57,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:58,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:49:59,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:49:59,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:00,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:01,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:02,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:03,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:04,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:50:04,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:05,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:06,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:07,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:08,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:09,117 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:50:09,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:10,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:50:10,013 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:50:10,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:11,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:12,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:13,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:14,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:50:14,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:15,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:15,631 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:50:16,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:17,657 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:50:18,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:19,777 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:50:24,778 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:50:25,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:50:26,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:26,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:50:26,707 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:50:28,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:29,875 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:50:34,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:50:34,876 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:50:35,409 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:36,221 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:39,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:50:43,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:50:43,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:43,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:50:43,451 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:50:44,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:45,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:50:45,632 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:50:45,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:50,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:50:52,226 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:50:53,391 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:54,755 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:50:55,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:51:00,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:51:01,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:51:02,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:03,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:51:03,423 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:51:04,469 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:06,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:51:11,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:51:15,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:51:15,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:15,636 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:51:15,642 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:16,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:51:16,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:16,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:51:16,900 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:51:17,642 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:18,063 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:51:18,642 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:19,642 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:20,642 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:21,642 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:22,043 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:51:22,642 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:23,642 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:24,643 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:25,643 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:26,643 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:27,043 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:51:27,643 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:28,643 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:29,643 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:30,643 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:31,643 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:31,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:51:31,901 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:51:32,643 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:33,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:51:33,644 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:34,644 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:35,644 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:36,644 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:37,644 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:38,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:51:38,644 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:39,644 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:40,644 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:41,644 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:42,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:43,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:51:43,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:44,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:45,640 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:51:45,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:46,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:46,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:51:46,901 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:51:47,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:48,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:51:48,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:49,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:50,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:51,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:52,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:53,057 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:51:53,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:54,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:55,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:56,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:57,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:58,057 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:51:58,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:51:59,716 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:00,716 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:01,716 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:01,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:52:01,901 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:52:02,716 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:03,716 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:04,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:04,716 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:05,716 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:06,716 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:07,716 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:08,717 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:09,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:09,717 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:10,717 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:11,717 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:12,717 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:13,717 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:14,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:15,643 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:52:19,646 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:24,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:29,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:33,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:52:33,586 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:52:33,586 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:33,587 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:52:34,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:34,671 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:35,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:36,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:37,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:38,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:39,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:39,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:40,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:41,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:42,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:43,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:44,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:44,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:45,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:45,647 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:52:46,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:47,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:48,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:48,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:52:48,550 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:52:49,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:49,707 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:50,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:51,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:52,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:53,548 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:54,548 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:54,707 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:55,548 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:56,548 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:57,548 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:58,548 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:59,548 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:59,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:00,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:01,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:02,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:03,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:03,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:53:03,550 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:53:04,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:04,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:05,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:06,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:07,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:08,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:09,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:09,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:10,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:11,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:12,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:13,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:14,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:14,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:15,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:15,651 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:53:16,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:17,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:18,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:53:18,550 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:53:18,590 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:19,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:20,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:20,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:21,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:22,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:23,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:24,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:25,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:25,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:26,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:27,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:28,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:29,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:30,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:30,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:31,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:32,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:33,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:53:33,550 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:53:33,590 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:34,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:35,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:36,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:36,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:37,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:38,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:39,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:40,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:41,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:41,618 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:42,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:43,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:44,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:45,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:45,654 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:53:46,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:46,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:47,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:48,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:53:48,550 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:53:48,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:49,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:50,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:51,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:51,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:52,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:53,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:54,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:55,555 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:56,555 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:56,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:57,555 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:58,555 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:59,555 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:00,555 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:01,555 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:01,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:02,555 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:03,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:54:03,551 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:54:03,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:04,556 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:05,556 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:06,556 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:06,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:07,556 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:08,556 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:09,556 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:10,556 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:11,556 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:11,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:12,557 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:13,557 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:14,557 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:15,557 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:15,657 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:54:16,557 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:17,557 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:17,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:18,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:54:18,551 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:54:18,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:19,557 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:20,557 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:21,557 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:22,558 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:22,707 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:23,558 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:24,558 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:25,558 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:26,558 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:27,558 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:27,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:32,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:37,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:42,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:45,661 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:54:48,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:53,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:58,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:03,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:08,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:10,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:55:10,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:10,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:55:10,208 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:55:11,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:12,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:13,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:13,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:14,145 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:55:14,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:15,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:15,665 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:55:16,147 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:55:16,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:17,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:18,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:19,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:19,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:20,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:21,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:22,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:23,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:24,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:24,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:25,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:55:25,168 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:55:25,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:26,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:26,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:55:26,385 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:55:26,386 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:55:26,388 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:55:27,154 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:55:27,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:28,156 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:55:28,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:29,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:30,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:30,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:31,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:32,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:33,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:34,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:35,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:35,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:36,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:37,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:38,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:39,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:39,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:55:39,717 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:55:39,717 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:55:39,719 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:55:40,163 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:55:40,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:55:40,168 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:55:40,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:41,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:41,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:42,165 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:55:42,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:43,404 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:44,390 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:45,390 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:45,668 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:55:46,390 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:46,671 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:47,390 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:48,390 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:49,390 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:50,390 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:51,390 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:51,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:52,391 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:53,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:55:53,153 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:55:53,154 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:55:53,156 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:55:53,172 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:55:53,391 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:54,173 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:55:54,391 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:55,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:55:55,168 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:55:55,464 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:56,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:57,320 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:57,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:58,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:59,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:00,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:01,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:02,321 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:02,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:03,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:04,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:05,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:06,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:56:06,301 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:56:06,301 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:56:06,302 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:56:06,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:07,184 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:56:07,341 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:07,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:08,185 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:56:08,609 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:09,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:10,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:56:10,168 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:56:10,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:11,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:12,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:13,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:13,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:14,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:15,607 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:15,672 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:56:16,607 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:17,607 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:18,607 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:18,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:19,607 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:19,670 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:56:19,672 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:56:19,672 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:56:19,674 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:56:20,193 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:56:20,607 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:21,607 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:22,194 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:56:22,607 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:23,607 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:23,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:24,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:25,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:56:25,168 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:56:25,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:26,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:27,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:28,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:29,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:29,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:30,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:31,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:32,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:33,037 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:56:33,039 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:56:33,039 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:56:33,041 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:56:33,202 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:56:33,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:34,203 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:56:34,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:35,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:35,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:36,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:37,713 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:38,713 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:39,713 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:40,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:40,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:56:40,168 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:56:40,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:41,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:42,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:43,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:44,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:45,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:45,676 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:56:45,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:46,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:56:46,421 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:56:46,421 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:56:46,423 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:56:46,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:47,212 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:56:47,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:48,213 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:56:48,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:49,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:50,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:50,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:51,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:52,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:53,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:54,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:55,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:56:55,168 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:56:55,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:56,316 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:56,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:57,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:58,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:59,718 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:56:59,720 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:56:59,720 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:56:59,722 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:56:59,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:00,222 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:57:00,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:01,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:57:01,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:02,224 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:57:02,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:03,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:04,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:05,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:06,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:57:06,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:07,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:08,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:09,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:10,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:57:10,169 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:57:10,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:11,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:12,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:57:12,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:13,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:57:13,202 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:57:13,202 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:13,204 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:13,231 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:57:13,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:14,233 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:57:14,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:15,680 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:57:15,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:16,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:17,682 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:57:17,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:18,878 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:19,878 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:20,878 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:21,878 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:22,683 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:57:22,878 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:23,878 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:24,879 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:25,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:57:25,169 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:57:25,879 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:26,590 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:57:26,591 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:57:26,592 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:26,594 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:26,879 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:27,242 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:57:27,879 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:28,243 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:57:28,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:57:28,928 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:29,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:30,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:31,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:32,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:33,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:57:33,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:34,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:35,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:36,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:37,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:38,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:57:38,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:39,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:39,953 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:57:39,955 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:57:39,955 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:39,957 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:40,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:57:40,189 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:57:40,252 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:57:40,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:41,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:42,254 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:57:42,918 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:43,918 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:44,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:57:45,005 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:45,683 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:57:45,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:46,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:47,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:48,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:49,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:57:49,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:50,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:51,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:52,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:53,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:57:53,249 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:57:53,250 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:53,251 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:53,262 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:57:53,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:54,262 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:57:54,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:55,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:57:55,187 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:57:55,276 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:57:55,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:56,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:57,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:58,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:59,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:00,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:58:00,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:01,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:02,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:03,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:04,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:05,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:58:05,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:06,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:58:06,599 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:58:06,600 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:58:06,602 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:58:06,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:07,273 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:58:07,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:08,274 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:58:08,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:09,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:10,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:58:10,289 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:58:10,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:58:10,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:11,994 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:12,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:13,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:14,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:15,352 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:58:15,687 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:58:15,994 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:16,994 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:17,994 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:18,994 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:19,919 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:58:19,921 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:58:19,921 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:58:19,923 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:58:20,159 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:20,282 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:58:20,961 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:58:21,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:22,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:22,284 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:58:23,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:24,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:25,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:25,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:58:25,272 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:58:26,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:26,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:58:27,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:28,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:29,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:30,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:31,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:31,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:58:32,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:33,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:33,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:58:33,208 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:58:33,209 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:58:33,210 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:58:33,292 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:58:34,057 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:34,293 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:58:35,057 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:36,057 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:37,057 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:37,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:58:38,057 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:39,057 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:40,057 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:40,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:58:40,272 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:58:41,057 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:42,058 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:42,392 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:58:43,058 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:44,058 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:45,058 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:45,691 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:58:46,058 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:46,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:58:46,428 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:58:46,429 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:58:46,430 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:58:47,058 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:47,301 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:58:47,468 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:58:48,058 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:48,302 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:58:49,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:50,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:51,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:52,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:52,468 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:58:53,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:54,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:55,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:55,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:58:55,272 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:58:56,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:57,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:58,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:58,435 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:58:59,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:58:59,789 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:58:59,791 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:58:59,791 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:58:59,793 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:59:00,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:00,310 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:59:01,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:02,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:02,312 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:59:03,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:03,831 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:59:04,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:05,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:06,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:07,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:08,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:08,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:59:09,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:10,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:10,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:59:10,272 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:59:11,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:12,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:13,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:13,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:59:13,196 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:59:13,197 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:59:13,199 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:59:13,320 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:59:14,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:14,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:59:14,321 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:59:15,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:15,694 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:59:16,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:17,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:18,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:19,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:19,697 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:59:20,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:21,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:22,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:23,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:24,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:24,698 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:59:25,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:25,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:59:25,272 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:59:26,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:26,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:59:26,620 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:59:26,621 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:59:26,623 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:59:27,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:27,331 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:59:28,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:28,332 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:59:29,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:30,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:30,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:59:31,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:32,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:33,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:34,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:35,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:35,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:59:36,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:37,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:38,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:39,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:40,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:40,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:59:40,123 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:59:40,123 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:59:40,125 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:59:40,341 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:59:40,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:59:40,363 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:59:41,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:41,525 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:59:42,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:42,342 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:59:43,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:44,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:45,235 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:45,698 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 22:59:46,219 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:46,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:59:47,219 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:48,219 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:49,219 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:50,219 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:51,220 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:51,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:59:52,220 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:53,220 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:53,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:59:53,551 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 22:59:53,551 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:59:53,553 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:59:54,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:54,351 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 22:59:54,352 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 22:59:55,220 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:55,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:59:55,362 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:59:56,221 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:57,235 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:57,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:59:58,233 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:59:59,233 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:00,233 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:01,233 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:02,234 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:02,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:00:03,234 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:04,234 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:05,234 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:06,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:06,876 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:00:06,878 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:00:06,878 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:00:06,880 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:00:07,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:07,360 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:00:07,919 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:00:08,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:08,361 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:00:09,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:10,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:10,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:00:10,362 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:00:11,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:12,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:13,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:13,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:00:14,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:15,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:15,702 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:00:16,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:17,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:18,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:18,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:00:19,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:20,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:00:20,013 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:00:20,013 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:00:20,015 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:00:20,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:20,376 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:00:21,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:22,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:22,378 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:00:23,317 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:24,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:00:24,318 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:25,318 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:25,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:00:25,362 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:00:26,318 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:27,318 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:28,318 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:29,318 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:29,464 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:00:30,318 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:31,318 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:32,318 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:33,318 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:33,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:00:33,364 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:00:33,365 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:00:33,367 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:00:33,384 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:00:34,319 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:34,386 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:00:35,319 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:35,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:00:36,319 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:37,319 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:38,389 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:39,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:40,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:00:40,362 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:00:40,403 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:40,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:00:41,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:42,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:43,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:44,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:45,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:45,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:00:45,706 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:00:46,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:46,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:00:46,575 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:00:46,575 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:00:46,577 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:00:47,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:47,394 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:00:48,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:48,396 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:00:49,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:50,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:50,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:00:51,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:52,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:53,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:54,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:55,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:00:55,363 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:00:55,403 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:56,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:56,480 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:00:57,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:58,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:59,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:00:59,719 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:00:59,721 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:00:59,721 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:00:59,723 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:01:00,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:00,404 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:01:00,405 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:01:01,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:01,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:01:02,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:03,404 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:04,403 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:05,404 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:06,404 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:06,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:01:07,404 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:08,404 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:09,404 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:10,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:01:10,363 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:01:10,404 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:11,404 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:12,404 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:12,466 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:01:12,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:01:12,991 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:01:12,992 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:01:12,994 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:01:13,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:13,413 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:01:14,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:14,414 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:01:15,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:15,709 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:01:16,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:17,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:17,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:01:18,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:19,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:20,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:21,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:22,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:22,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:01:23,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:24,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:25,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:01:25,363 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:01:25,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:26,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:01:26,205 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:01:26,205 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:01:26,207 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:01:26,423 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:01:26,435 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:27,433 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:28,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:01:28,424 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:01:28,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:29,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:30,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:31,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:32,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:33,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:01:33,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:34,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:35,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:36,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:37,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:38,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:01:38,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:39,483 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:01:39,484 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:01:39,485 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:01:39,486 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:01:39,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:40,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:01:40,363 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:01:40,433 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:01:40,433 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:01:40,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:41,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:42,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:43,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:01:43,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:44,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:45,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:45,714 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:01:46,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:47,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:48,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:48,716 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:01:49,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:50,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:51,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:52,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:01:52,621 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:01:52,622 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:01:52,623 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:01:52,861 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:53,441 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:01:53,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:54,446 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:01:54,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:54,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:01:55,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:01:55,363 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:01:55,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:56,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:57,796 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:58,653 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:01:59,653 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:00,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:02:00,653 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:01,653 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:02,654 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:03,654 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:04,654 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:05,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:02:05,654 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:05,925 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:02:05,927 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:02:05,927 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:02:05,929 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:02:06,455 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:02:06,654 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:07,654 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:08,654 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:09,457 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:02:09,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:10,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:02:10,364 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:02:10,654 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:11,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:02:11,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:12,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:13,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:14,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:15,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:15,717 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:02:16,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:16,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:02:17,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:18,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:19,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:02:19,012 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:02:19,013 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:02:19,014 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:02:19,464 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:02:19,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:20,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:21,466 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:02:21,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:22,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:02:22,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:23,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:24,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:25,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:02:25,363 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:02:25,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:26,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:27,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:02:27,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:28,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:29,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:30,657 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:31,657 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:32,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:02:32,159 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:02:32,160 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:02:32,162 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:02:32,473 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:02:32,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:33,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:02:33,657 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:34,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:35,476 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:02:35,657 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:36,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:37,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:38,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:02:38,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:39,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:40,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:02:40,364 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:02:40,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:41,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:42,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:43,432 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:02:43,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:44,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:45,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:02:45,378 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:02:45,378 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:02:45,380 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:02:45,483 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:02:45,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:45,720 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:02:46,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:47,485 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:02:47,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:48,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:48,722 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:02:49,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:50,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:51,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:52,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:53,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:53,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:02:54,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:55,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:02:55,364 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:02:55,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:56,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:57,694 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:58,671 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:02:58,672 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:02:58,673 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:02:58,674 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:02:58,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:02:59,493 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:02:59,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:02:59,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:00,996 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:01,495 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:03:01,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:02,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:03,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:04,713 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:03:04,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:05,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:06,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:07,994 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:08,994 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:09,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:03:09,994 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:10,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:03:10,364 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:03:10,994 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:11,823 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:03:11,825 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:03:11,826 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:03:11,827 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:03:12,063 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:12,503 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:03:13,060 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:13,504 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:03:14,142 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:14,865 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:03:15,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:15,724 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:03:16,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:17,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:18,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:19,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:20,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:20,726 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:03:21,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:22,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:23,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:24,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:25,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:03:25,103 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:03:25,104 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:03:25,105 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:03:25,342 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:25,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:03:25,364 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:03:25,513 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:03:26,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:26,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:03:27,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:27,515 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:03:28,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:29,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:30,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:31,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:31,525 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:03:32,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:33,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:34,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:35,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:36,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:36,526 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:03:37,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:38,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:38,445 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:03:38,446 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:03:38,446 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:03:38,448 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:03:38,521 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:03:39,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:39,522 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:03:40,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:40,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:03:40,364 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:03:41,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:42,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:42,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:03:43,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:44,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:45,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:45,726 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:03:46,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:47,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:47,729 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:03:48,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:49,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:50,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:51,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:51,657 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:03:51,658 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:03:51,659 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:03:51,660 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:03:52,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:52,531 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:03:53,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:53,532 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:03:53,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:03:54,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:55,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:55,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:03:55,364 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:03:56,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:57,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:58,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:59,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:03:59,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:04:00,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:01,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:02,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:03,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:04,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:04,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:04:05,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:04:05,103 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:04:05,104 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:04:05,106 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:04:05,339 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:05,539 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:04:06,339 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:07,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:07,541 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:04:08,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:09,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:10,143 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:04:10,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:04:10,365 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:04:10,419 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:11,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:12,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:13,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:14,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:15,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:15,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:04:15,731 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:04:16,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:17,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:18,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:18,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:04:18,565 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:04:18,565 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:04:18,567 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:04:19,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:19,548 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:04:19,549 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:04:20,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:20,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:04:21,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:22,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:23,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:24,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:25,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:04:25,365 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:04:25,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:26,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:26,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:04:27,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:28,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:29,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:30,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:31,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:31,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:04:31,876 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:04:31,878 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:04:31,878 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:04:31,880 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:04:32,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:32,557 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:04:33,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:33,558 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:04:34,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:35,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:36,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:36,918 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:04:37,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:38,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:39,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:40,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:04:40,365 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:04:40,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:41,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:42,432 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:04:42,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:43,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:44,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:45,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:04:45,173 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:04:45,173 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:04:45,175 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:04:45,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:45,566 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:04:45,734 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:04:46,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:47,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:47,568 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:04:47,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:04:48,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:49,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:50,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:51,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:52,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:52,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:04:53,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:54,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:55,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:04:55,365 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:04:55,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:56,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:57,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:58,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:04:58,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:58,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:04:58,597 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:04:58,598 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:04:58,599 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:04:59,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:04:59,576 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:04:59,576 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:05:00,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:01,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:02,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:03,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:05:03,638 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:04,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:05,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:06,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:07,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:08,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:08,638 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:05:09,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:10,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:05:10,365 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:05:10,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:11,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:11,790 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:05:11,792 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:05:11,792 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:05:11,794 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:05:12,584 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:05:12,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:13,584 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:05:13,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:13,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:05:14,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:15,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:15,736 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:05:16,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:17,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:18,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:19,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:19,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:05:20,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:21,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:22,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:23,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:24,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:24,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:05:24,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:05:24,988 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:05:24,989 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:05:24,990 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:05:25,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:05:25,367 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:05:25,593 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:05:25,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:26,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:27,595 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:05:27,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:28,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:29,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:30,527 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:05:30,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:31,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:32,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:33,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:34,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:35,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:05:35,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:36,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:37,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:38,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:05:38,260 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:05:38,260 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:05:38,262 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:05:38,606 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:05:38,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:39,607 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:05:39,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:40,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:05:40,366 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:05:40,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:05:40,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:41,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:42,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:43,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:44,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:45,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:05:45,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:45,740 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:05:46,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:47,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:48,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:49,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:50,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:50,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:05:51,555 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:05:51,556 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:05:51,556 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:05:51,558 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:05:51,615 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:05:51,790 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:52,789 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:53,617 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:05:53,868 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:54,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:55,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:05:55,366 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:05:55,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:56,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:05:56,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:57,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:58,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:05:59,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:00,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:01,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:06:01,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:02,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:03,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:04,854 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:06:04,856 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:06:04,856 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:06:04,858 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:06:04,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:05,625 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:06:05,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:06,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:06,896 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:06:07,626 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:06:07,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:08,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:09,898 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:10,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:06:10,366 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:06:10,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:11,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:12,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:06:12,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:13,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:14,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:15,744 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:06:15,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:16,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:17,746 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:06:17,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:18,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:06:18,023 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:06:18,023 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:06:18,025 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:06:18,633 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:06:18,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:19,634 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:06:19,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:20,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:21,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:22,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:23,064 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:06:23,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:24,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:25,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:06:25,366 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:06:25,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:26,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:27,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:28,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:06:28,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:29,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:30,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:31,373 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:06:31,374 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:06:31,375 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:06:31,376 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:06:31,641 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:06:31,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:32,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:33,643 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:06:33,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:34,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:06:34,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:35,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:36,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:37,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:38,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:39,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:06:39,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:40,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:06:40,367 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:06:40,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:41,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:42,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:43,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:44,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:06:44,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:06:44,678 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:06:44,679 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:06:44,681 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:06:44,925 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:45,651 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:06:45,651 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:06:45,747 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:06:45,924 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:47,008 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:48,002 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:49,002 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:49,750 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:06:50,002 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:51,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:52,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:53,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:54,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:54,750 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:06:55,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:55,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:06:55,367 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:06:56,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:57,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:58,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:58,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:06:58,025 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:06:58,026 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:06:58,027 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:06:58,660 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:06:59,004 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:06:59,661 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:07:00,004 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:00,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:07:01,004 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:02,004 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:03,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:04,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:05,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:07:05,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:06,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:07,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:08,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:09,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:10,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:07:10,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:10,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:07:10,367 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:07:11,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:11,221 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:07:11,223 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:07:11,224 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:07:11,225 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:07:11,669 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:07:12,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:13,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:13,670 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:07:14,133 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:15,133 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:15,263 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:07:15,751 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:07:16,133 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:17,133 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:18,133 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:19,133 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:20,134 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:20,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:07:21,134 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:22,134 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:23,134 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:24,134 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:24,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:07:24,514 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:07:24,515 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:07:24,516 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:07:24,678 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:07:25,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:25,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:07:25,367 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:07:25,679 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:07:26,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:26,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:07:27,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:28,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:29,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:30,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:31,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:31,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:07:32,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:33,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:34,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:35,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:36,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:36,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:07:37,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:37,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:07:37,767 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:07:37,768 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:07:37,769 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:07:38,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:38,688 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:07:39,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:39,689 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:07:40,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:40,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:07:40,367 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:07:41,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:41,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:07:42,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:43,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:44,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:45,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:45,755 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:07:46,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:46,757 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:07:47,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:48,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:49,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:50,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:51,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:07:51,091 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:07:51,092 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:07:51,094 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:07:51,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:51,696 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:07:52,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:07:52,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:53,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:53,699 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:07:54,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:55,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:55,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:07:55,367 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:07:56,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:57,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:57,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:07:58,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:07:59,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:00,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:01,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:02,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:02,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:08:03,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:04,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:04,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:08:04,477 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:08:04,478 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:08:04,479 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:08:04,706 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:08:05,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:05,707 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:08:06,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:07,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:07,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:08:08,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:09,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:10,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:10,367 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:08:10,367 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:08:11,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:12,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:13,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:13,436 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:08:14,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:15,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:15,759 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:08:16,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:17,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:17,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:08:17,533 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:08:17,533 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:08:17,535 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:08:17,715 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:08:18,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:18,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:08:19,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:19,717 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:08:20,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:21,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:22,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:23,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:23,574 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:08:24,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:25,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:25,367 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:08:25,367 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:08:26,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:27,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:28,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:29,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:29,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:08:30,232 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:30,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:08:30,634 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:08:30,635 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:08:30,637 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:08:30,725 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:08:31,234 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:31,726 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:08:32,233 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:33,233 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:34,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:34,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:08:35,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:36,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:37,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:38,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:39,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:39,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:08:40,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:40,367 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:08:40,368 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:08:41,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:42,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:43,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:43,934 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:08:43,936 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:08:43,936 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:08:43,938 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:08:44,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:44,735 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:08:44,976 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:08:45,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:45,736 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:08:45,762 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:08:46,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:47,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:48,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:49,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:50,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:50,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:08:51,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:52,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:53,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:54,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:55,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:55,367 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:08:55,368 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:08:56,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:56,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:08:57,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:08:57,197 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:08:57,198 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:08:57,200 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:08:57,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:57,745 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:08:58,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:59,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:08:59,747 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:09:00,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:01,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:02,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:09:02,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:03,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:04,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:05,329 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:06,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:07,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:09:07,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:08,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:09,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:10,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:10,367 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:09:10,368 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:09:10,582 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:09:10,584 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:09:10,584 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:09:10,586 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:09:10,760 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:09:11,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:11,762 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:09:12,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:12,587 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:09:13,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:14,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:15,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:15,766 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:09:16,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:17,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:17,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:09:18,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:19,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:20,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:21,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:22,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:22,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:09:23,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:23,900 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:09:23,901 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:09:23,902 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:09:23,903 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:09:24,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:24,774 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:09:25,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:25,367 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:09:25,368 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:09:25,775 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:09:26,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:27,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:28,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:28,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:09:29,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:30,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:31,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:32,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:33,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:33,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:09:34,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:35,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:36,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:37,309 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:09:37,311 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:09:37,311 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:09:37,313 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:09:37,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:37,783 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:09:38,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:39,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:09:39,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:39,785 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:09:40,367 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:09:40,368 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:09:40,614 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:41,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:42,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:43,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:44,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:09:44,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:45,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:45,770 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:09:46,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:47,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:48,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:49,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:49,773 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:09:50,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:50,682 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:09:50,683 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:09:50,684 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:09:50,686 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:09:50,792 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:09:51,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:51,793 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:09:52,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:53,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:54,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:55,368 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:09:55,368 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:09:55,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:09:55,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:56,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:57,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:58,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:09:59,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:00,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:10:00,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:01,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:02,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:03,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:03,862 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:10:03,864 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:10:03,865 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:10:03,866 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:10:04,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:04,802 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:10:05,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:05,803 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:10:05,904 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:10:06,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:07,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:08,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:09,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:10,368 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:10:10,369 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:10:10,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:11,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:10:11,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:12,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:13,707 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:14,707 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:15,707 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:15,773 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:10:16,707 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:16,776 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:10:17,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:10:17,129 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:10:17,130 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:10:17,131 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:10:17,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:17,811 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:10:18,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:19,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:19,813 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:10:20,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:21,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:22,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:10:22,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:23,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:24,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:25,368 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:10:25,369 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:10:25,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:26,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:27,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:10:27,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:28,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:29,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:30,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:10:30,449 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:10:30,450 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:10:30,452 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:10:30,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:30,821 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:10:31,817 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:31,822 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:10:32,490 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:10:32,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:33,817 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:34,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:35,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:36,817 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:37,491 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:10:37,817 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:38,817 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:39,817 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:40,368 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:10:40,369 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:10:40,817 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:41,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:42,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:10:42,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:43,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:43,826 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:10:43,827 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:10:43,828 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:10:43,830 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:10:44,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:44,830 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:10:45,776 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:10:45,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:45,830 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:10:46,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:47,779 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:10:47,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:48,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:49,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:50,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:51,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:52,779 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:10:52,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:53,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:54,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:55,368 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:10:55,369 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:10:55,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:56,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:57,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:10:57,013 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:10:57,013 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:10:57,015 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:10:57,839 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:10:57,840 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:10:57,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:58,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:10:58,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:10:59,911 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:00,911 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:01,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:02,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:03,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:11:03,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:04,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:05,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:06,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:07,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:08,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:11:08,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:09,912 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:10,288 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:11:10,290 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:11:10,290 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:11:10,292 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:11:10,525 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:11:10,526 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:11:10,848 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:11:10,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:11,849 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:11:11,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:12,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:13,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:11:13,962 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:14,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:15,780 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:11:15,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:16,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:17,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:18,782 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:11:18,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:19,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:20,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:21,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:22,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:23,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:11:23,445 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:11:23,446 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:11:23,447 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:11:23,857 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:11:23,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:24,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:11:24,949 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:25,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:11:25,424 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:11:25,859 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:11:25,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:26,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:27,949 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:28,949 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:29,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:11:29,949 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:30,949 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:31,949 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:32,949 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:33,949 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:34,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:11:34,949 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:35,949 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:36,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:11:36,690 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:11:36,691 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:11:36,693 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:11:36,867 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:11:36,950 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:37,868 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:11:37,950 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:39,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:39,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:11:40,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:40,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:11:40,424 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:11:41,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:42,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:43,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:44,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:45,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:45,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:11:45,784 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:11:46,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:47,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:48,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:49,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:49,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:11:49,885 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:11:49,886 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:11:49,887 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:11:50,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:50,878 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:11:50,924 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:11:51,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:51,879 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:11:52,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:53,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:54,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:55,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:55,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:11:55,424 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:11:56,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:56,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:11:57,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:58,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:11:59,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:00,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:01,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:01,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:12:02,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:03,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:03,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:12:03,244 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:12:03,245 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:12:03,247 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:12:03,887 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:12:04,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:05,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:05,889 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:12:06,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:07,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:07,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:12:08,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:09,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:10,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:10,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:12:10,424 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:12:11,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:12,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:12,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:12:13,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:14,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:15,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:15,789 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:12:16,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:16,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:12:16,540 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:12:16,541 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:12:16,542 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:12:16,896 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:12:17,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:17,581 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:12:17,897 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:12:18,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:19,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:20,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:21,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:22,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:22,581 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:12:23,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:24,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:25,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:25,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:12:25,424 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:12:26,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:27,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:28,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:28,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:12:29,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:29,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:12:29,743 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:12:29,744 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:12:29,745 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:12:29,905 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:12:30,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:31,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:31,908 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:12:32,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:33,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:33,783 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:12:34,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:35,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:36,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:37,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:38,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:38,784 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:12:39,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:40,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:40,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:12:40,425 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:12:41,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:42,043 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:42,874 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:12:42,876 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:12:42,877 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:12:42,878 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:12:42,915 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:12:43,113 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:43,916 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:12:43,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:12:44,112 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:45,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:45,792 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:12:46,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:47,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:48,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:49,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:49,794 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:12:50,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:51,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:52,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:53,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:54,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:54,795 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:12:55,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:55,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:12:55,425 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:12:56,111 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:12:56,113 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:12:56,113 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:12:56,114 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:12:56,355 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:56,924 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:12:57,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:57,925 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:12:58,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:12:59,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:00,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:13:00,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:01,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:02,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:03,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:04,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:05,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:13:05,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:06,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:07,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:08,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:09,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:09,507 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:13:09,509 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:13:09,509 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:13:09,511 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:13:09,932 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:13:10,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:10,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:13:10,425 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:13:10,506 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:13:11,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:11,934 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:13:12,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:13,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:14,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:15,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:15,507 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:13:15,794 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:13:16,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:17,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:18,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:19,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:20,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:20,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:13:21,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:22,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:22,824 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:13:22,825 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:13:22,826 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:13:22,827 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:13:22,940 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:13:23,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:23,941 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:13:24,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:25,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:25,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:13:25,425 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:13:26,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:26,559 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:13:27,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:28,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:29,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:30,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:31,260 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:31,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:13:32,260 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:33,260 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:34,260 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:35,260 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:36,119 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:13:36,121 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:13:36,121 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:13:36,123 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:13:36,359 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:36,950 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:13:37,161 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:13:37,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:37,950 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:13:38,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:39,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:40,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:40,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:13:40,425 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:13:41,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:42,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:42,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:13:43,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:44,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:45,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:45,798 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:13:46,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:47,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:47,801 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:13:48,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:49,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:49,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:13:49,310 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:13:49,311 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:13:49,312 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:13:49,958 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:13:49,958 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:13:50,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:51,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:52,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:53,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:53,350 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:13:54,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:55,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:55,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:13:55,425 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:13:56,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:57,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:58,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:13:58,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:13:59,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:00,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:01,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:02,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:02,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:14:02,570 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:14:02,570 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:14:02,572 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:14:02,966 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:14:03,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:03,610 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:14:03,967 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:14:04,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:05,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:06,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:07,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:08,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:08,610 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:14:09,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:10,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:10,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:14:10,425 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:14:11,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:12,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:13,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:14,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:14,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:14:15,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:15,802 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:14:15,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:14:15,893 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:14:15,894 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:14:15,895 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:14:15,976 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:14:16,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:17,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:17,977 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:14:18,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:19,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:19,934 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:14:20,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:21,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:22,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:23,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:24,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:24,935 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:14:25,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:25,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:14:25,426 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:14:26,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:27,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:28,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:29,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:14:29,231 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:14:29,232 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:14:29,233 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:14:29,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:29,986 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:14:29,987 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:14:30,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:14:30,370 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:31,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:32,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:33,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:34,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:35,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:14:35,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:36,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:37,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:38,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:39,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:40,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:14:40,372 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:40,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:14:40,426 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:14:41,372 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:42,372 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:42,482 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:14:42,483 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:14:42,484 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:14:42,486 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:14:42,996 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:14:43,372 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:43,996 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:14:44,372 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:45,372 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:45,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:14:45,805 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:14:46,372 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:47,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:48,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:49,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:50,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:50,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:14:51,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:52,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:53,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:54,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:55,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:14:55,426 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:14:55,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:55,640 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:14:55,642 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:14:55,642 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:14:55,644 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:14:56,004 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:14:56,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:56,681 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:14:57,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:58,006 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:14:58,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:14:59,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:00,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:01,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:01,682 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:15:02,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:03,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:04,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:05,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:06,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:06,683 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:15:07,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:08,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:08,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:15:08,821 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:15:08,821 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:15:08,824 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:15:09,013 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:15:09,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:10,014 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:15:10,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:15:10,426 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:15:10,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:11,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:12,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:12,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:15:13,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:14,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:15,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:15,808 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:15:16,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:17,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:17,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:15:18,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:19,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:20,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:21,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:22,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:15:22,198 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:15:22,198 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:15:22,200 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:15:22,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:23,023 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:15:23,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:15:23,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:24,024 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:15:24,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:25,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:15:25,426 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:15:25,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:26,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:27,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:28,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:28,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:15:29,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:30,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:31,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:32,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:33,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:33,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:15:34,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:35,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:35,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:15:35,532 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:15:35,533 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:15:35,534 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:15:36,033 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:15:36,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:37,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:38,035 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:15:38,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:38,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:15:39,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:40,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:15:40,426 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:15:40,646 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:41,646 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:42,646 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:43,577 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:15:43,646 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:44,646 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:45,646 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:45,812 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:15:46,646 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:47,646 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:48,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:48,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:15:49,040 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:15:49,041 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:15:49,042 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:15:49,044 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:15:49,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:50,044 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:15:50,045 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:15:50,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:51,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:52,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:53,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:54,082 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:15:54,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:55,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:15:55,426 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:15:55,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:56,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:57,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:58,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:15:59,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:15:59,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:00,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:01,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:02,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:16:02,337 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:16:02,338 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:16:02,339 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:16:02,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:03,053 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:16:03,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:04,054 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:16:04,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:05,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:16:05,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:06,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:07,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:08,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:09,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:10,378 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:16:10,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:16:10,427 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:16:10,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:11,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:12,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:13,680 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:14,680 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:15,579 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:16:15,588 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:16:15,589 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:16:15,590 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:16:15,591 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:16:15,815 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:16:15,875 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:16,062 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:16:16,865 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:17,934 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:18,064 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:16:18,934 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:19,934 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:20,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:16:20,934 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:21,934 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:22,934 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:23,934 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:24,934 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:25,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:16:25,427 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:16:25,935 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:26,540 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:16:26,934 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:27,935 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:28,773 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:16:28,775 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:16:28,775 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:16:28,777 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:16:29,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:29,072 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:16:30,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:30,072 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:16:31,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:31,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:16:32,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:33,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:34,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:35,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:36,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:36,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:16:37,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:38,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:39,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:40,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:40,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:16:40,427 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:16:41,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:41,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:16:41,985 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:16:41,985 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:16:41,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:16:41,987 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:16:42,080 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:16:42,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:43,222 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:44,081 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:16:44,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:45,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:45,819 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:16:46,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:47,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:47,821 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:16:48,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:49,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:50,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:51,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:52,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:52,822 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:16:53,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:54,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:55,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:16:55,241 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:16:55,242 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:16:55,244 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:16:55,478 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:16:55,479 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:16:55,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:56,089 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:16:56,089 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:16:56,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:57,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:58,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:16:58,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:16:59,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:00,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:01,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:02,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:03,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:03,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:17:04,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:05,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:06,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:07,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:08,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:17:08,523 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:17:08,524 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:17:08,526 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:17:08,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:09,098 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:17:09,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:17:09,760 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:10,099 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:17:10,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:17:10,478 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:17:10,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:11,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:12,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:13,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:14,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:17:14,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:15,821 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:17:15,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:16,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:17,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:18,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:19,824 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:17:19,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:20,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:21,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:17:21,764 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:17:21,765 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:17:21,767 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:17:21,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:22,108 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:17:22,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:23,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:24,110 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:17:24,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:25,483 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:17:25,484 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:17:25,587 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:17:25,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:26,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:27,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:28,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:29,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:30,588 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:17:30,842 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:31,842 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:32,842 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:33,842 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:34,842 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:35,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:17:35,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:36,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:37,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:38,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:39,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:40,482 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:17:40,483 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:17:40,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:41,574 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:17:41,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:42,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:43,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:44,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:45,825 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:17:45,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:46,827 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:17:46,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:48,127 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:17:48,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:52,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:17:55,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 23:17:56,894 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:17:57,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:17:57,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:17:57,837 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:18:02,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:18:04,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 23:18:04,469 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:05,907 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:07,960 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:18:12,961 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:18:13,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 23:18:14,695 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:18:14,696 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:18:15,149 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:15,829 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:18:18,831 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:18:21,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 23:18:21,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:23,072 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:23,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:18:28,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:18:30,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 23:18:30,317 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:31,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:18:31,241 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:18:32,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:34,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:18:39,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 23:18:39,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:18:43,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:43,581 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:44,492 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:18:44,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:44,583 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:18:45,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:18:45,103 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:18:45,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:45,832 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:18:46,493 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:18:46,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:47,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:48,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:49,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:49,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:18:50,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:51,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:52,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:53,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:54,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:54,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:18:55,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:56,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:57,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:58,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:59,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:18:59,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:19:00,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:19:00,102 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:19:00,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:01,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:02,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:03,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:04,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:05,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:19:05,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:06,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:07,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:08,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:09,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:10,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:19:10,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:11,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:12,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:13,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:14,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:15,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:19:15,103 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:19:15,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:15,836 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:19:15,842 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:19:16,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:17,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:18,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:19,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:20,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:20,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:19:21,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:22,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:23,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:24,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:25,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:25,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:19:26,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:27,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:28,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:29,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:30,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:19:30,103 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:19:30,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:31,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:19:31,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:32,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:33,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:34,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:35,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:36,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:19:36,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:37,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:38,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:39,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:40,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:19:41,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:19:45,839 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:19:46,842 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:19:51,842 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:19:56,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:20:01,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:20:02,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 23:20:02,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:20:02,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:02,055 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:20:03,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:04,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:05,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:06,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:07,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:07,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:20:08,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:09,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:10,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:11,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:12,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:12,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:20:13,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:14,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:15,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:15,843 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:20:16,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:17,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:20:17,014 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:20:17,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:17,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:20:18,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:19,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:20,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:21,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:22,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:22,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:20:23,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:24,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:25,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:26,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:27,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:27,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:20:28,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:29,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:30,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:31,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:32,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:20:32,014 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:20:32,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:33,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:33,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:20:34,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:35,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:36,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:37,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:38,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:38,125 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:20:39,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:40,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:41,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:42,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:43,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:43,125 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:20:44,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:45,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:45,847 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:20:46,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:47,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:20:47,014 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:20:47,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:48,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:49,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:49,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:20:50,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:51,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:52,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:53,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:54,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:54,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:20:55,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:56,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:57,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:58,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:59,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:20:59,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:21:00,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:01,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:02,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:21:02,015 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:21:02,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:03,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:04,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:04,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:21:05,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:06,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:07,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:08,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:09,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:09,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:21:10,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:11,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:12,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:13,026 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:14,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:14,165 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:21:15,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:15,850 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:21:16,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:17,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:21:17,015 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:21:17,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:18,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:19,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:19,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:21:20,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:21,027 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:22,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:23,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:24,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:24,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:21:25,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:26,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:27,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:28,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:29,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:29,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:21:30,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:31,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:32,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:21:32,015 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:21:32,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:33,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:34,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:35,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:35,100 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:21:36,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:37,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:38,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:39,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:40,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:40,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:21:41,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:42,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:43,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:44,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:45,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:45,101 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:21:45,854 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:21:46,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:47,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:21:47,015 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:21:47,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:48,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:49,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:50,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:50,161 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:21:51,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:52,032 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:53,032 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:54,032 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:55,032 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:21:55,161 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:22:00,162 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:22:05,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:22:10,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:22:15,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:22:15,857 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:22:20,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:22:25,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:22:30,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:22:35,861 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:22:36,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 23:22:36,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:22:36,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:36,376 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:22:37,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:38,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:39,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:40,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:40,481 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:22:41,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:41,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:22:42,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:43,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:44,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:45,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:45,859 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:22:46,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:46,862 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:22:47,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:48,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:49,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:50,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:51,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:22:51,335 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:22:51,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:52,215 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:22:52,216 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:22:52,217 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:22:52,217 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:22:52,218 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:22:52,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:52,490 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:22:53,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:54,492 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:22:54,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:55,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:56,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:57,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:22:57,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:58,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:22:59,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:00,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:01,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:02,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:23:02,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:03,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:04,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:05,490 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:23:05,492 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:23:05,492 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:23:05,494 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:23:05,500 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:23:05,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:06,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:23:06,335 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:23:06,505 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:23:06,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:07,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:23:07,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:08,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:09,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:10,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:11,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:12,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:23:12,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:13,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:14,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:15,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:15,863 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:23:16,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:17,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:17,866 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:23:18,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:18,867 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:23:18,869 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:23:18,869 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:23:18,871 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:23:19,517 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:23:19,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:20,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:21,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:23:21,335 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:23:21,519 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:23:21,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:22,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:23,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:23:23,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:24,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:25,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:26,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:27,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:28,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:23:28,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:29,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:30,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:31,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:32,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:23:32,230 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:23:32,230 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:23:32,232 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:23:32,528 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:23:32,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:33,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:34,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:23:34,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:35,530 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:23:35,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:36,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:23:36,336 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:23:36,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:37,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:38,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:39,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:23:39,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:40,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:41,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:42,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:43,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:44,480 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:23:44,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:45,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:45,772 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:23:45,774 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:23:45,774 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:23:45,776 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:23:45,867 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:23:46,539 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:23:46,639 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:47,540 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:23:47,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:48,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:49,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:49,870 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:23:50,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:51,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:23:51,336 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:23:51,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:52,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:53,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:54,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:55,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:23:55,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:56,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:57,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:58,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:23:59,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:23:59,365 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:23:59,365 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:23:59,367 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:23:59,549 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:23:59,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:00,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:01,404 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:24:01,551 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:24:01,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:02,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:03,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:04,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:05,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:06,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:24:06,336 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:24:06,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:24:06,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:07,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:08,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:09,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:10,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:11,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:24:11,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:12,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:24:12,637 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:24:12,637 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:24:12,639 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:24:12,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:13,561 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:24:13,863 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:14,957 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:15,562 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:24:15,870 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:24:15,944 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:16,872 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:24:16,944 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:17,944 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:18,944 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:19,944 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:20,944 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:21,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:24:21,336 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:24:21,945 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:22,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:24:22,945 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:23,945 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:24,945 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:25,783 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:24:25,784 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:24:25,785 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:24:25,787 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:24:26,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:26,572 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:24:27,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:27,573 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:24:27,824 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:24:28,146 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:29,134 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:30,134 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:31,134 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:32,134 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:32,825 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:24:33,134 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:34,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:35,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:36,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:36,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:24:36,336 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:24:37,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:38,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:38,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:24:39,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:39,159 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:24:39,160 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:24:39,161 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:24:39,162 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:24:39,582 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:24:40,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:41,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:41,584 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:24:42,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:43,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:44,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:24:44,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:45,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:45,873 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:24:46,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:47,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:48,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:49,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:49,876 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:24:50,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:51,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:51,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:24:51,336 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:24:52,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:52,506 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:24:52,507 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:24:52,508 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:24:52,510 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:24:52,593 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:24:53,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:54,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:55,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:55,548 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:24:55,595 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:24:56,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:57,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:58,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:24:59,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:00,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:00,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:25:01,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:02,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:03,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:04,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:05,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:05,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:25:05,645 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:25:05,646 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:25:05,647 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:25:05,649 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:25:06,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:06,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:25:06,336 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:25:06,604 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:25:07,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:07,605 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:25:08,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:09,319 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:10,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:11,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:11,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:25:12,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:13,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:14,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:15,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:15,877 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:25:16,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:16,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:25:17,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:18,300 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:18,893 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:25:18,894 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:25:18,895 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:25:18,896 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:25:19,300 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:19,614 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:25:20,300 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:21,300 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:21,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:25:21,336 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:25:21,616 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:25:22,300 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:22,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:25:23,300 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:24,300 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:25,300 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:26,300 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:27,300 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:27,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:25:28,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:29,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:30,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:31,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:32,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:25:32,170 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:25:32,171 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:25:32,172 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:25:32,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:32,625 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:25:33,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:25:33,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:33,626 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:25:34,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:35,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:36,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:36,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:25:36,336 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:25:37,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:38,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:38,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:25:39,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:40,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:41,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:42,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:43,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:43,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:25:44,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:45,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:45,607 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:25:45,609 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:25:45,609 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:25:45,611 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:25:45,635 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:25:45,881 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:25:46,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:47,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:47,637 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:25:48,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:48,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:25:49,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:50,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:51,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:51,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:25:51,337 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:25:52,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:53,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:54,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:54,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:25:55,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:56,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:57,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:58,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:58,871 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:25:58,873 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:25:58,873 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:25:58,875 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:25:59,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:25:59,648 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:25:59,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:26:00,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:01,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:01,650 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:26:02,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:03,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:04,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:04,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:26:05,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:06,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:06,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:26:06,337 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:26:07,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:08,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:09,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:10,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:10,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:26:11,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:12,126 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:26:12,128 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:26:12,129 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:26:12,130 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:26:12,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:12,658 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:26:13,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:13,660 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:26:14,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:15,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:15,884 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:26:15,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:26:16,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:17,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:18,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:19,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:20,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:20,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:26:21,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:26:21,337 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:26:21,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:22,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:23,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:24,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:25,418 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:26:25,421 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:26:25,421 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:26:25,423 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:26:25,657 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:25,669 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:26:26,459 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:26:26,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:27,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:27,671 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:26:28,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:29,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:30,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:31,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:26:31,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:32,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:33,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:34,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:35,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:36,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:26:36,338 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:26:36,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:37,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:26:37,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:38,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:38,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:26:38,702 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:26:38,703 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:26:38,705 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:26:39,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:39,682 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:26:39,682 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:26:40,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:41,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:42,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:42,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:26:43,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:44,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:45,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:45,888 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:26:46,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:47,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:47,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:26:48,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:49,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:50,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:51,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:26:51,337 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:26:51,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:51,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:26:51,892 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:26:51,892 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:26:51,894 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:26:52,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:52,691 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:26:52,931 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:26:53,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:53,692 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:26:54,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:55,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:56,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:57,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:57,932 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:26:58,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:26:59,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:00,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:01,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:02,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:02,933 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:27:03,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:04,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:05,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:27:05,207 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:27:05,207 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:27:05,209 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:27:05,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:05,701 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:27:06,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:27:06,340 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:27:06,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:07,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:07,703 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:27:08,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:27:08,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:09,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:10,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:11,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:12,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:13,409 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:27:13,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:14,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:15,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:15,892 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:27:16,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:17,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:18,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:18,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:27:18,606 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:27:18,607 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:27:18,607 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:27:18,608 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:27:18,712 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:27:19,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:19,713 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:27:20,574 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:21,339 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:27:21,340 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:27:21,574 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:22,574 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:23,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:24,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:27:24,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:25,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:26,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:27,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:28,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:29,447 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:27:29,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:30,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:31,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:31,961 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:27:31,962 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:27:31,963 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:27:31,964 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:27:32,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:32,724 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:27:33,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:33,725 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:27:34,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:35,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:27:35,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:36,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:27:36,340 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:27:36,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:37,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:38,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:39,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:40,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:27:40,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:41,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:42,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:43,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:44,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:45,219 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:27:45,220 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:27:45,221 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:27:45,223 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:27:45,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:45,735 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:27:45,896 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:27:45,902 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:27:46,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:47,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:47,737 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:27:48,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:49,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:50,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:50,898 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:27:51,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:27:51,340 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:27:51,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:52,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:53,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:54,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:55,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:56,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:27:56,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:57,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:58,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:27:58,553 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:27:58,554 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:27:58,555 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:27:58,745 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:27:58,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:27:59,747 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:27:59,783 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:00,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:01,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:28:01,855 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:02,855 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:03,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:04,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:05,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:06,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:28:06,341 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:28:06,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:07,433 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:28:07,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:08,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:09,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:10,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:11,779 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:28:11,780 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:28:11,781 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:28:11,782 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:28:12,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:12,757 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:28:12,821 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:28:13,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:13,758 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:28:14,093 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:15,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:15,899 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:28:16,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:17,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:17,902 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:28:18,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:19,081 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:20,081 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:21,081 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:21,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:28:21,341 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:28:22,081 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:23,081 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:23,409 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:28:24,081 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:25,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:28:25,003 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:28:25,003 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:28:25,005 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:28:25,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:25,768 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:28:26,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:27,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:27,770 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:28:28,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:29,043 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:28:29,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:30,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:31,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:32,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:33,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:34,044 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:28:34,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:35,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:36,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:36,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:28:36,341 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:28:37,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:38,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:38,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:28:38,274 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:28:38,274 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:28:38,276 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:28:38,778 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:28:39,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:39,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:28:39,779 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:28:40,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:41,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:42,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:43,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:44,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:44,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:28:45,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:45,903 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:28:46,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:47,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:48,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:49,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:49,906 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:28:50,139 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:51,139 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:51,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:28:51,341 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:28:51,608 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:28:51,610 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:28:51,611 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:28:51,612 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:28:51,790 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:28:52,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:53,139 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:53,791 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:28:54,139 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:55,139 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:55,614 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:28:56,139 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:57,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:58,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:28:59,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:00,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:00,614 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:29:01,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:02,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:03,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:04,140 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:04,963 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:29:04,965 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:29:04,966 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:29:04,967 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:29:05,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:05,801 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:29:05,802 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:29:06,005 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:29:06,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:06,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:29:06,341 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:29:07,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:08,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:09,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:10,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:11,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:11,506 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:29:12,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:13,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:14,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:15,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:15,908 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:29:16,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:16,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:29:17,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:18,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:18,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:29:18,333 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:29:18,334 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:29:18,335 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:29:18,812 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:29:19,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:19,813 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:29:20,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:21,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:21,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:29:21,341 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:29:22,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:22,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:29:23,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:24,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:25,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:26,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:27,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:27,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:29:28,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:29,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:30,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:31,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:31,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:29:31,551 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:29:31,552 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:29:31,553 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:29:31,822 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:29:32,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:32,591 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:29:33,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:33,824 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:29:34,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:35,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:36,341 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:29:36,341 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:29:36,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:37,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:38,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:38,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:29:39,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:40,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:41,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:42,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:43,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:43,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:29:44,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:44,831 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:29:44,832 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:29:44,833 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:29:44,834 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:29:45,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:45,835 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:29:45,835 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:29:45,910 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:29:46,350 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:47,350 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:48,350 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:48,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:29:49,350 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:50,350 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:51,341 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:29:51,341 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:29:51,381 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:52,350 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:53,350 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:54,350 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:54,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:29:55,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:56,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:57,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:58,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:58,370 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:29:58,371 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:29:58,372 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:29:58,373 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:29:58,848 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:29:59,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:29:59,850 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:30:00,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:00,413 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:30:01,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:02,351 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:03,466 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:04,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:05,413 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:30:05,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:06,341 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:30:06,342 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:30:06,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:07,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:08,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:09,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:10,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:10,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:30:11,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:11,822 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:30:11,824 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:30:11,824 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:30:11,826 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:30:11,859 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:30:12,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:13,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:13,861 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:30:14,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:15,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:15,864 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:30:15,914 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:30:16,507 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:17,507 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:18,507 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:19,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:20,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:20,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:30:21,341 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:30:21,342 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:30:21,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:22,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:23,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:24,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:25,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:30:25,290 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:30:25,291 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:30:25,292 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:30:25,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:25,871 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:30:26,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:30:26,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:27,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:27,873 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:30:28,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:29,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:30,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:31,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:30:31,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:32,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:33,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:34,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:35,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:36,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:30:36,342 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:30:36,342 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:30:36,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:37,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:38,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:38,828 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:30:38,830 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:30:38,830 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:30:38,832 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:30:38,881 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:30:39,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:39,883 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:30:40,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:41,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:41,870 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:30:42,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:43,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:44,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:45,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:45,918 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:30:46,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:46,920 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:30:47,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:48,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:49,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:50,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:51,342 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:30:51,342 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:30:51,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:52,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:30:52,242 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:30:52,242 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:30:52,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:30:52,244 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:30:52,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:52,893 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:30:53,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:53,894 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:30:54,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:55,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:56,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:57,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:30:57,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:58,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:30:59,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:00,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:01,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:02,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:31:02,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:03,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:04,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:05,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:31:05,525 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:31:05,526 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:31:05,527 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:31:05,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:05,904 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:31:06,342 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:31:06,342 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:31:06,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:07,459 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:31:07,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:07,906 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:31:08,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:09,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:10,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:11,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:12,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:31:12,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:13,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:14,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:15,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:15,922 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:31:16,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:17,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:17,924 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:31:18,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:18,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:31:18,772 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:31:18,772 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:31:18,773 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:31:18,914 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:31:19,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:19,915 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:31:20,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:21,342 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:31:21,342 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:31:21,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:22,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:23,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:31:23,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:24,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:25,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:26,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:27,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:28,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:31:28,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:29,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:30,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:31,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:32,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:31:32,015 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:31:32,015 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:31:32,017 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:31:32,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:32,924 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:31:33,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:33,925 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:31:34,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:31:34,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:35,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:36,342 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:31:36,343 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:31:36,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:37,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:38,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:39,435 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:31:39,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:40,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:41,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:42,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:43,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:44,435 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:31:44,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:45,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:31:45,449 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:31:45,450 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:31:45,451 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:31:45,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:45,926 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:31:45,934 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:31:46,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:47,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:47,935 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:31:48,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:49,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:49,928 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:31:50,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:51,342 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:31:51,343 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:31:51,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:52,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:53,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:54,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:55,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:31:55,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:56,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:57,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:58,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:58,868 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:31:58,869 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:31:58,870 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:31:58,871 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:31:58,944 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:31:59,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:31:59,945 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:32:00,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:00,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:32:01,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:02,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:03,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:04,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:05,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:05,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:32:06,343 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:32:06,343 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:32:06,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:07,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:08,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:09,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:10,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:11,467 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:32:11,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:12,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:32:12,197 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:32:12,198 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:32:12,200 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:32:12,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:12,955 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:32:13,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:13,956 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:32:14,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:15,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:15,929 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:32:16,866 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:16,931 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:32:17,866 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:18,866 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:19,866 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:20,866 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:21,342 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:32:21,343 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:32:21,866 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:22,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:32:22,866 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:23,866 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:24,866 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:25,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:32:25,519 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:32:25,519 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:32:25,521 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:32:25,869 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:25,965 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:32:26,869 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:27,559 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:32:27,918 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:27,967 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:32:28,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:29,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:30,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:31,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:32,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:32:32,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:33,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:34,918 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:35,918 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:36,343 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:32:36,343 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:32:36,918 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:37,918 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:38,471 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:32:38,755 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:32:38,756 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:32:38,757 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:32:38,759 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:32:38,976 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:32:38,996 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:39,976 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:32:39,994 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:41,074 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:42,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:43,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:43,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:32:44,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:45,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:45,932 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:32:46,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:47,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:48,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:48,935 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:32:49,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:50,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:51,060 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:51,343 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:32:51,343 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:32:51,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:32:51,960 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:32:51,961 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:32:51,962 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:32:51,985 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:32:52,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:53,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:53,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:32:53,988 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:32:54,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:55,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:56,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:57,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:58,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:32:58,965 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:32:59,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:00,102 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:01,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:02,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:03,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:03,965 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:33:04,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:05,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:05,414 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:33:05,415 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:33:05,416 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:33:05,417 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:33:05,996 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:33:06,103 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:06,343 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:33:06,343 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:33:07,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:07,998 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:33:08,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:09,113 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:09,430 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:33:10,111 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:11,112 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:12,112 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:13,112 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:14,112 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:14,431 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:33:15,112 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:15,936 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:33:16,112 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:17,112 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:18,112 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:18,773 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:33:18,774 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:33:18,775 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:33:18,776 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:33:19,006 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:33:19,113 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:19,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:33:20,007 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:33:20,113 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:21,113 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:21,343 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:33:21,343 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:33:22,113 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:23,113 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:24,113 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:25,113 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:25,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:33:26,113 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:27,113 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:28,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:29,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:30,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:30,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:33:31,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:32,078 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:33:32,079 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:33:32,080 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:33:32,081 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:33:32,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:33,017 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:33:33,325 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:34,018 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:33:34,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:35,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:36,119 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:33:36,343 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:33:36,343 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:33:36,404 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:37,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:38,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:39,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:40,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:41,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:41,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:33:42,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:43,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:44,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:45,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:33:45,331 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:33:45,332 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:33:45,333 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:33:45,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:45,940 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:33:46,027 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:33:46,028 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:33:46,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:46,942 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:33:47,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:48,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:49,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:50,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:51,343 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:33:51,344 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:33:51,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:52,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:33:52,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:53,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:54,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:55,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:56,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:57,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:33:57,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:58,649 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:33:58,651 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:33:58,652 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:33:58,653 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:33:58,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:33:59,038 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:33:59,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:00,039 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:34:00,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:01,967 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:02,691 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:34:02,967 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:03,967 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:04,967 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:05,967 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:06,343 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:34:06,344 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:34:06,968 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:07,968 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:08,482 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:34:08,968 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:09,968 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:10,968 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:11,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:34:11,959 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:34:11,960 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:34:11,961 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:34:12,049 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:34:12,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:13,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:13,999 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:34:14,051 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:34:14,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:15,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:15,944 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:34:16,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:17,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:18,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:19,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:19,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:34:20,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:21,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:21,343 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:34:21,344 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:34:22,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:23,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:24,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:25,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:34:25,258 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:34:25,258 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:34:25,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:34:25,260 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:34:25,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:26,061 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:34:26,061 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:34:26,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:27,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:28,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:29,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:30,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:34:30,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:31,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:32,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:33,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:34,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:35,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:34:35,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:36,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:34:36,344 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:34:36,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:37,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:38,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:38,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:34:38,636 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:34:38,637 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:34:38,639 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:34:39,072 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:34:39,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:40,073 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:34:40,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:40,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:34:41,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:42,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:43,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:44,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:45,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:45,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:34:45,947 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:34:46,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:47,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:48,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:49,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:50,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:50,950 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:34:51,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:34:51,344 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:34:51,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:51,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:34:51,838 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:34:51,839 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:34:51,840 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:34:52,082 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:34:52,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:53,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:54,084 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:34:54,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:55,401 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:56,401 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:56,878 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:34:57,401 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:58,401 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:34:59,401 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:00,401 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:01,401 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:01,879 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:35:02,401 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:03,401 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:04,401 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:05,071 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:35:05,073 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:35:05,073 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:35:05,075 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:35:05,093 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:35:05,402 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:06,095 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:35:06,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:35:06,345 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:35:06,401 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:07,402 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:07,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:35:08,402 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:09,402 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:10,402 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:11,402 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:12,402 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:12,510 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:35:13,402 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:14,403 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:15,403 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:15,951 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:35:16,403 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:17,403 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:17,953 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:35:18,221 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:35:18,222 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:35:18,223 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:35:18,224 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:35:18,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:19,106 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:35:19,457 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:20,107 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:35:20,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:21,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:35:21,345 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:35:21,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:22,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:23,447 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:35:23,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:24,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:25,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:26,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:27,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:28,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:35:28,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:29,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:30,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:31,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:31,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:35:31,678 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:35:31,678 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:35:31,680 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:35:32,116 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:35:32,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:33,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:33,719 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:35:34,120 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:35:34,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:35,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:36,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:35:36,345 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:35:36,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:37,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:38,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:39,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:35:39,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:40,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:41,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:42,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:43,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:44,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:35:44,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:44,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:35:44,966 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:35:44,967 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:35:44,968 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:35:45,129 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:35:45,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:45,954 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:35:46,130 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:35:46,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:47,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:48,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:49,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:49,957 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:35:50,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:51,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:35:51,345 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:35:51,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:52,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:53,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:54,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:55,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:35:55,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:56,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:57,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:58,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:35:58,272 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:35:58,272 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:35:58,274 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:35:58,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:35:59,140 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:35:59,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:00,142 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:36:00,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:01,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:36:01,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:02,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:03,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:04,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:05,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:06,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:36:06,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:36:06,345 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:36:06,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:07,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:08,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:09,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:10,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:11,433 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:36:11,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:36:11,569 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:36:11,569 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:36:11,571 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:36:11,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:12,151 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:36:12,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:13,894 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:14,153 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:36:14,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:15,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:15,958 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:36:16,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:16,960 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:36:17,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:18,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:19,878 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:20,878 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:21,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:36:21,345 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:36:21,878 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:22,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:36:22,878 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:23,878 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:24,842 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:36:24,844 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:36:24,844 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:36:24,846 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:36:25,070 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:25,162 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:36:26,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:26,163 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:36:27,157 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:27,885 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:36:28,149 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:29,149 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:30,149 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:31,149 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:32,149 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:32,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:36:33,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:34,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:35,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:36,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:36,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:36:36,346 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:36:37,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:38,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:38,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:36:38,365 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:36:38,365 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:36:38,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:36:38,367 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:36:39,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:39,173 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:36:40,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:40,174 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:36:41,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:42,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:43,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:43,405 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:36:44,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:45,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:45,962 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:36:46,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:47,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:48,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:48,965 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:36:49,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:50,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:51,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:51,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:36:51,346 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:36:51,748 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:36:51,750 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:36:51,751 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:36:51,752 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:36:52,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:52,182 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:36:53,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:54,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:54,184 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:36:54,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:36:55,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:56,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:57,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:58,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:59,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:36:59,755 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:37:00,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:01,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:02,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:03,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:04,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:04,756 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:37:05,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:05,157 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:37:05,158 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:37:05,159 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:37:05,160 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:37:05,193 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:37:06,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:06,194 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:37:06,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:37:06,346 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:37:07,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:08,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:09,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:10,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:10,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:37:11,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:12,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:13,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:14,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:15,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:15,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:37:15,966 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:37:16,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:17,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:18,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:18,463 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:37:18,464 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:37:18,465 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:37:18,466 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:37:19,204 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:37:19,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:20,205 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:37:20,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:20,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:37:21,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:21,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:37:21,345 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:37:22,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:23,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:24,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:25,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:26,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:26,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:37:27,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:28,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:29,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:30,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:31,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:31,480 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:37:31,604 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:37:31,605 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:37:31,606 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:37:31,607 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:37:32,216 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:37:32,216 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:37:32,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:33,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:34,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:35,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:36,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:36,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:37:36,346 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:37:36,483 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:37:37,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:38,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:39,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:40,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:41,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:41,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:37:42,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:43,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:44,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:44,933 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:37:44,934 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:37:44,934 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:37:44,936 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:37:45,226 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:37:45,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:45,969 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:37:46,227 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:37:46,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:46,971 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:37:47,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:48,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:49,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:50,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:51,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:51,345 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:37:51,346 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:37:52,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:52,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:37:53,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:54,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:55,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:56,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:57,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:57,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:37:58,157 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:37:58,158 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:37:58,159 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:37:58,160 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:37:58,235 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:37:58,393 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:37:59,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:00,237 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:38:00,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:01,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:02,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:03,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:38:03,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:04,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:05,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:06,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:06,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:38:06,346 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:38:07,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:08,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:08,465 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:38:09,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:10,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:11,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:11,412 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:38:11,414 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:38:11,414 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:38:11,416 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:38:12,248 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:38:12,249 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:38:12,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:13,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:14,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:14,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:38:15,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:15,973 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:38:16,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:17,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:18,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:19,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:19,975 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:38:20,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:21,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:21,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:38:21,347 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:38:22,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:23,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:24,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:24,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:38:24,732 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:38:24,732 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:38:24,734 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:38:25,258 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:38:25,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:25,773 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:38:26,259 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:38:26,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:27,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:28,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:29,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:30,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:30,773 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:38:31,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:32,296 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:33,296 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:34,296 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:35,296 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:35,774 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:38:36,296 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:36,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:38:36,347 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:38:37,296 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:38,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:38:38,122 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:38:38,123 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:38:38,125 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:38:38,269 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:38:38,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:39,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:40,271 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:38:40,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:41,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:38:41,430 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:42,430 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:43,431 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:44,431 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:45,431 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:45,977 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:38:46,431 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:46,979 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:38:47,431 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:48,431 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:49,431 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:50,431 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:51,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:38:51,293 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:38:51,294 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:38:51,295 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:38:51,575 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:51,576 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:38:51,576 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:38:52,281 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:38:52,281 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:38:52,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:52,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:38:53,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:54,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:55,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:56,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:57,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:57,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:38:58,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:38:59,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:00,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:01,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:02,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:02,736 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:39:03,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:04,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:04,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:39:04,725 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:39:04,726 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:39:04,727 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:39:05,291 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:39:05,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:06,292 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:39:06,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:39:06,427 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:39:06,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:07,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:08,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:39:08,623 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:09,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:10,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:11,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:12,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:13,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:39:13,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:14,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:15,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:15,980 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:39:16,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:17,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:17,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:39:17,959 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:39:17,960 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:39:17,961 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:39:18,301 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:39:18,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:18,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:39:19,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:20,303 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:39:20,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:21,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:39:21,427 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:39:21,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:22,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:23,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:24,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:39:24,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:25,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:26,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:27,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:28,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:29,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:39:29,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:30,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:31,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:39:31,212 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:39:31,213 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:39:31,215 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:39:31,312 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:39:31,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:32,313 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:39:32,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:33,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:34,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:35,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:39:35,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:36,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:39:36,427 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:39:36,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:37,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:38,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:39,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:40,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:39:40,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:41,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:42,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:43,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:44,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:39:44,568 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:39:44,569 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:39:44,570 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:39:44,809 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:45,323 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:39:45,610 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:39:45,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:45,984 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:39:46,324 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:39:46,896 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:47,894 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:48,894 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:49,894 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:50,894 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:50,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:39:51,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:39:51,428 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:39:51,894 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:52,894 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:53,894 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:54,894 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:55,894 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:56,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:39:56,895 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:57,854 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:39:57,855 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:39:57,855 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:39:57,856 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:39:58,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:58,333 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:39:58,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:39:59,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:00,335 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:40:00,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:01,895 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:40:01,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:02,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:03,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:04,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:05,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:06,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:40:06,428 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:40:06,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:07,519 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:40:07,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:08,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:09,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:10,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:11,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:40:11,275 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:40:11,275 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:40:11,277 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:40:11,344 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:40:11,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:12,345 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:40:12,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:13,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:40:13,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:14,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:15,988 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:40:15,994 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:16,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:17,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:18,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:40:18,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:19,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:20,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:21,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:40:21,428 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:40:21,991 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:22,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:23,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:24,585 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:40:24,586 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:40:24,586 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:40:24,587 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:40:24,587 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:40:24,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:25,356 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:40:25,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:26,357 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:40:26,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:27,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:28,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:29,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:40:29,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:30,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:31,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:32,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:33,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:34,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:40:34,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:35,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:36,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:40:36,428 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:40:36,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:37,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:40:37,835 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:40:37,836 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:40:37,837 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:40:38,074 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:38,367 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:40:38,368 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:40:39,070 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:39,876 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:40:40,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:41,147 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:42,147 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:43,148 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:44,148 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:44,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:40:45,148 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:45,991 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:40:46,148 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:47,148 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:48,148 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:49,148 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:49,993 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:40:50,149 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:51,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:40:51,109 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:40:51,109 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:40:51,111 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:40:51,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:51,378 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:40:51,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:40:51,428 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:40:52,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:52,379 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:40:53,431 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:54,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:55,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:55,614 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:40:56,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:57,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:58,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:40:59,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:00,421 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:00,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:41:01,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:02,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:03,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:04,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:04,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:41:04,430 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:41:04,430 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:41:04,432 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:41:05,388 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:41:05,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:06,392 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:41:06,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:06,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:41:06,428 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:41:06,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:41:07,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:08,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:09,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:10,494 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:11,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:11,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:41:12,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:13,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:14,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:15,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:15,995 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:41:16,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:16,997 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:41:17,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:17,795 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:41:17,797 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:41:17,798 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:41:17,799 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:41:18,401 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:41:18,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:19,402 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:41:19,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:20,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:21,480 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:41:21,481 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:41:21,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:22,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:22,639 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:41:23,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:24,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:25,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:26,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:27,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:27,640 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:41:28,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:29,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:30,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:31,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:41:31,117 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:41:31,117 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:41:31,119 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:41:31,412 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:41:31,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:32,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:33,157 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:41:33,414 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:41:33,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:34,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:35,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:36,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:41:36,461 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:41:36,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:37,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:38,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:38,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:41:39,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:40,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:41,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:42,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:43,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:43,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:41:44,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:41:44,379 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:41:44,379 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:41:44,381 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:41:44,422 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:41:44,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:45,513 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:45,998 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:41:46,513 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:47,425 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:41:47,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:48,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:49,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:41:49,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:50,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:51,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:41:51,461 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:41:51,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:52,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:53,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:54,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:54,603 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:41:55,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:56,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:57,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:57,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:41:57,621 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:41:57,622 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:41:57,623 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:41:58,434 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:41:58,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:59,435 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:41:59,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:41:59,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:42:00,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:01,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:02,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:03,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:04,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:04,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:42:05,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:06,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:42:06,462 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:42:06,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:07,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:08,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:09,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:10,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:10,614 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:42:11,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:42:11,056 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:42:11,056 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:42:11,058 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:42:11,445 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:42:11,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:12,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:13,447 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:42:13,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:14,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:15,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:16,002 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:42:16,009 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:42:16,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:17,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:18,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:19,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:20,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:21,005 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:42:21,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:42:21,462 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:42:21,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:22,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:23,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:24,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:42:24,386 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:42:24,387 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:42:24,389 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:42:24,455 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:42:24,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:25,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:26,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:42:26,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:27,458 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:42:27,707 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:28,707 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:29,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:30,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:31,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:42:31,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:32,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:33,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:34,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:35,708 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:36,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:42:36,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:42:36,462 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:42:36,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:37,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:42:37,660 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:42:37,661 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:42:37,662 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:42:37,894 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:38,470 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:42:38,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:39,471 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:42:39,980 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:40,968 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:41,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:42:41,969 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:42,968 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:43,968 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:44,968 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:45,968 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:46,006 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:42:46,969 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:47,008 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:42:47,969 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:48,969 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:49,969 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:50,789 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:42:50,791 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:42:50,791 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:42:50,793 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:42:51,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:51,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:42:51,462 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:42:51,480 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:42:52,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:52,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:42:53,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:53,482 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:42:54,094 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:55,095 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:56,094 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:57,094 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:57,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:42:58,094 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:42:59,095 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:00,095 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:01,095 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:02,095 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:02,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:43:03,095 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:04,040 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:43:04,042 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:43:04,043 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:43:04,044 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:43:04,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:04,491 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:43:05,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:05,492 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:43:06,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:06,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:43:06,462 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:43:07,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:07,614 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:43:08,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:09,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:10,174 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:11,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:12,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:12,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:43:13,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:14,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:15,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:16,009 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:43:16,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:17,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:17,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:43:17,363 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:43:17,363 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:43:17,366 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:43:17,501 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:43:18,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:18,403 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:43:19,175 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:19,503 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:43:20,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:21,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:21,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:43:21,462 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:43:22,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:23,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:23,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:43:24,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:25,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:26,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:27,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:28,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:28,551 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:43:29,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:30,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:30,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:43:30,690 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:43:30,690 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:43:30,692 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:43:31,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:31,513 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:43:32,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:33,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:33,514 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:43:33,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:43:34,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:35,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:36,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:36,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:43:36,462 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:43:37,177 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:38,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:39,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:39,555 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:43:40,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:41,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:42,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:43,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:43,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:43:43,949 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:43:43,950 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:43:43,951 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:43:44,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:44,523 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:43:44,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:43:45,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:45,524 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:43:46,013 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:43:46,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:47,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:48,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:49,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:50,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:43:50,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:51,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:51,461 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:43:51,462 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:43:52,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:53,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:54,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:55,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:55,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:43:56,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:57,149 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:43:57,151 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:43:57,151 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:43:57,153 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:43:57,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:57,534 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:43:58,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:59,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:43:59,536 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:44:00,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:01,191 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:44:01,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:02,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:03,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:04,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:05,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:06,192 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:44:06,283 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:06,462 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:44:06,462 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:44:07,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:08,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:09,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:10,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:10,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:44:10,599 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:44:10,600 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:44:10,601 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:44:11,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:11,544 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:44:11,640 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:44:12,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:13,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:13,546 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:44:14,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:15,291 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:16,016 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:44:16,291 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:17,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:44:17,291 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:18,291 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:19,291 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:20,291 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:21,291 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:21,462 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:44:21,462 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:44:22,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:22,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:44:23,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:23,898 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:44:23,900 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:44:23,900 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:44:23,902 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:44:24,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:24,554 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:44:25,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:25,555 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:44:26,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:27,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:27,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:44:28,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:29,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:30,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:31,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:32,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:32,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:44:33,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:34,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:35,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:36,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:36,462 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:44:36,463 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:44:37,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:44:37,138 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:44:37,138 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:44:37,140 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:44:37,382 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:37,564 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:44:38,141 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:44:38,379 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:39,460 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:39,565 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:44:40,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:41,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:42,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:43,142 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:44:43,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:44,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:45,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:46,021 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:44:46,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:47,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:48,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:49,024 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:44:49,445 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:50,445 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:50,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:44:50,452 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:44:50,452 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:44:50,454 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:44:50,574 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:44:51,445 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:51,462 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:44:51,462 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:44:52,445 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:53,445 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:53,576 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:44:54,445 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:54,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:44:55,445 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:56,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:57,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:58,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:59,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:44:59,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:45:00,469 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:01,468 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:02,468 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:03,468 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:04,468 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:04,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:45:05,468 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:06,462 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:45:06,463 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:45:06,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:07,468 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:08,469 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:09,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:10,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:10,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:45:11,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:12,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:13,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:14,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:15,476 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:15,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:45:16,024 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:45:17,368 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:17,593 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:45:21,061 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:45:24,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 23:45:24,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:25,590 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:26,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:45:26,546 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:45:26,546 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:45:31,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:45:32,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:34,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:36,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:45:41,548 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 23:45:41,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:45:42,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:42,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:45:42,598 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:45:43,214 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:46,027 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:45:47,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:45:50,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 23:45:50,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:52,030 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:45:52,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:45:57,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:45:59,006 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 23:45:59,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:01,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:01,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:46:01,293 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:46:02,370 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:46:07,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:46:09,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 23:46:12,371 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:46:12,507 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:12,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:13,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:13,729 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:46:14,007 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:46:14,007 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:46:14,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:15,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:15,730 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:46:16,031 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:46:16,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:17,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:18,033 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:46:18,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:19,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:20,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:21,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:22,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:23,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:46:23,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:24,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:25,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:26,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:27,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:28,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:46:28,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:29,007 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:46:29,007 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:46:29,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:30,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:31,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:32,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:33,136 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:46:33,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:34,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:35,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:36,513 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:37,513 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:38,137 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:46:38,513 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:39,607 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:40,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:41,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:42,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:43,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:46:43,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:44,007 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:46:44,008 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:46:44,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:45,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:46,034 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:46:46,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:47,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:48,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:49,037 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:46:49,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:50,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:51,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:52,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:53,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:54,037 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:46:54,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:55,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:56,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:57,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:58,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:46:59,007 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:46:59,008 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:46:59,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:46:59,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:00,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:01,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:02,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:03,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:04,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:47:04,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:05,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:06,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:07,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:09,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:47:14,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:47:16,038 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:47:20,040 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:47:25,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:47:30,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:47:30,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 23:47:30,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:47:30,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:30,282 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:47:31,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:32,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:33,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:34,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:35,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:35,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:47:36,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:37,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:38,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:39,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:40,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:40,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:47:41,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:42,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:43,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:44,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:45,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:45,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:47:45,244 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:47:45,392 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:47:46,041 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:47:46,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:47,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:48,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:49,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:50,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:51,044 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:47:51,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:52,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:53,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:54,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:55,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:56,045 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:47:56,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:57,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:58,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:47:59,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:00,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:00,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:48:00,245 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:48:01,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:01,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:48:02,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:03,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:04,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:05,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:06,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:06,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:48:07,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:08,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:09,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:10,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:11,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:11,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:48:12,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:13,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:14,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:15,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:48:15,245 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:48:15,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:16,048 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:48:16,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:17,051 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:48:17,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:18,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:19,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:20,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:21,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:22,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:48:22,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:23,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:24,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:25,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:26,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:27,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:48:27,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:28,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:29,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:30,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:48:30,245 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:48:30,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:31,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:32,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:32,392 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:48:33,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:34,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:35,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:36,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:37,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:37,393 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:48:38,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:39,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:40,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:41,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:42,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:42,394 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:48:43,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:44,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:45,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:48:45,245 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:48:45,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:46,048 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:48:46,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:47,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:48,051 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:48:48,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:49,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:50,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:51,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:52,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:53,051 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:48:53,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:54,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:55,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:56,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:57,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:58,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:48:58,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:48:59,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:00,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:49:00,245 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:49:00,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:01,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:02,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:03,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:03,320 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:49:04,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:05,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:06,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:07,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:08,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:08,321 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:49:09,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:10,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:11,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:12,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:13,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:13,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:49:14,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:15,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:49:15,245 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:49:15,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:16,052 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:49:16,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:17,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:18,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:19,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:49:19,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:20,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:21,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:22,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:23,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:49:24,055 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:49:29,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:49:34,056 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:49:39,057 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:49:44,058 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:49:46,055 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:49:50,058 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:49:55,058 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:50:00,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:50:05,060 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:50:05,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 23:50:05,469 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:05,469 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:50:05,470 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:50:06,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:07,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:08,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:09,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:09,581 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:50:10,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:10,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:50:11,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:11,583 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:50:12,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:13,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:14,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:15,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:15,789 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:50:16,059 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:50:16,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:17,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:18,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:19,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:20,430 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:20,430 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:50:20,431 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:50:21,389 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:50:21,391 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:50:21,392 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:50:21,392 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:50:21,393 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:50:21,590 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:50:21,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:22,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:23,592 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:50:23,717 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:24,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:25,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:26,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:50:26,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:27,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:28,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:29,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:30,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:31,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:50:31,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:32,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:33,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:34,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:35,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:50:35,195 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:50:35,195 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:50:35,197 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:50:35,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:50:35,430 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:50:35,602 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:50:35,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:36,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:50:36,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:37,603 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:50:37,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:38,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:39,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:40,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:41,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:50:41,710 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:42,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:43,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:44,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:45,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:46,063 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:50:46,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:47,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:50:47,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:48,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:48,973 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:50:48,975 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:50:48,976 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:50:48,977 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:50:49,613 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:50:49,614 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:50:49,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:50,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:50:50,430 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:50:50,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:51,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:52,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:50:52,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:53,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:54,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:55,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:56,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:57,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:50:57,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:58,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:50:59,713 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:00,713 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:01,713 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:02,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:51:02,670 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:51:02,672 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:51:02,673 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:51:02,674 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:51:02,897 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:03,624 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:51:03,624 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:51:03,809 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:04,809 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:05,430 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:51:05,430 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:51:05,809 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:06,809 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:07,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:51:07,809 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:08,809 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:09,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:10,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:11,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:12,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:51:12,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:13,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:14,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:15,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:16,068 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:51:16,226 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:51:16,228 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:51:16,228 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:51:16,230 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:51:16,633 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:51:16,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:17,634 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:51:17,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:18,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:51:18,811 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:19,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:20,430 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:51:20,430 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:51:20,896 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:21,897 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:22,897 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:23,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:51:23,897 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:24,897 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:25,897 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:26,897 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:27,897 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:28,590 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:51:28,897 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:29,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:51:29,744 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:51:29,745 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:51:29,746 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:51:29,983 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:30,644 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:51:30,982 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:31,645 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:51:32,044 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:33,033 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:33,784 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:51:34,033 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:35,033 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:35,430 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:51:35,430 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:51:36,033 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:37,033 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:38,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:39,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:39,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:51:40,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:41,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:42,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:43,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:43,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:51:43,486 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:51:43,487 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:51:43,488 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:51:43,654 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:51:44,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:45,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:45,526 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:51:45,655 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:51:46,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:46,069 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:51:47,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:48,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:49,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:50,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:50,430 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:51:50,431 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:51:50,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:51:51,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:52,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:53,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:54,069 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:55,069 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:55,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:51:56,069 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:57,052 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:51:57,053 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:51:57,054 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:51:57,055 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:51:57,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:57,664 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:51:57,665 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:51:58,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:51:59,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:00,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:01,093 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:52:01,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:02,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:03,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:04,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:05,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:05,430 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:52:05,431 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:52:06,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:06,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:52:07,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:08,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:09,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:10,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:10,646 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:52:10,647 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:52:10,648 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:52:10,649 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:52:10,674 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:52:11,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:11,675 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:52:11,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:52:12,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:13,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:14,330 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:15,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:16,073 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:52:16,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:17,075 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:52:17,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:18,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:19,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:20,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:20,430 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:52:20,431 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:52:21,316 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:22,316 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:22,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:52:23,316 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:24,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:52:24,301 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:52:24,301 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:52:24,303 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:52:24,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:24,684 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:52:25,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:25,685 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:52:26,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:27,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:28,341 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:52:28,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:29,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:30,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:31,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:32,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:33,342 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:52:33,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:34,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:35,430 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:52:35,431 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:52:35,471 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:36,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:37,439 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:37,999 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:52:38,000 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:52:38,001 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:52:38,002 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:52:38,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:38,696 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:52:39,040 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:52:39,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:39,697 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:52:40,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:41,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:42,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:43,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:44,040 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:52:44,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:45,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:46,077 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:52:46,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:47,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:48,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:49,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:52:49,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:50,431 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:52:50,431 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:52:50,471 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:51,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:51,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:52:51,567 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:52:51,567 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:52:51,569 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:52:51,706 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:52:52,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:53,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:53,707 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:52:54,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:54,608 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:52:55,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:56,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:57,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:58,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:59,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:52:59,609 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:53:00,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:01,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:02,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:03,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:04,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:04,610 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:53:05,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:53:05,339 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:53:05,339 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:53:05,340 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:53:05,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:05,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:53:05,596 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:53:05,717 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:53:06,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:07,483 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:07,718 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:53:08,483 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:09,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:09,760 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:53:10,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:11,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:12,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:13,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:14,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:14,761 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:53:15,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:16,081 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:53:16,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:17,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:18,484 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:19,037 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:53:19,039 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:53:19,039 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:53:19,041 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:53:19,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:19,727 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:53:19,728 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:53:20,079 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:53:20,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:20,592 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:53:20,593 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:53:21,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:22,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:23,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:24,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:25,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:25,691 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:53:26,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:27,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:28,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:29,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:30,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:30,692 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:53:31,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:32,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:32,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:53:32,623 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:53:32,624 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:53:32,625 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:53:32,735 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:53:33,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:33,736 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:53:34,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:35,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:35,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:53:35,594 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:53:36,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:36,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:53:37,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:38,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:39,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:40,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:41,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:41,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:53:42,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:43,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:44,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:45,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:46,085 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:53:46,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:53:46,428 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:53:46,428 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:53:46,430 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:53:46,669 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:46,745 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:53:47,468 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:53:47,559 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:47,746 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:53:48,559 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:49,559 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:50,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:50,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:53:50,593 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:53:51,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:52,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:52,682 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:53:53,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:54,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:55,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:56,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:57,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:57,683 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:53:58,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:53:59,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:00,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:54:00,199 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:54:00,200 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:54:00,202 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:54:00,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:00,754 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:54:01,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:01,755 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:54:02,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:03,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:54:03,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:04,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:05,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:05,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:54:05,595 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:54:06,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:07,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:08,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:08,663 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:54:09,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:10,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:11,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:12,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:13,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:13,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:54:13,811 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:54:13,813 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:54:13,813 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:54:13,815 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:54:14,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:14,765 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:54:15,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:15,766 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:54:16,088 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:54:16,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:17,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:18,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:19,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:54:19,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:20,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:20,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:54:20,594 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:54:21,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:22,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:23,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:24,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:24,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:54:25,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:26,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:27,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:54:27,548 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:54:27,549 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:54:27,551 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:54:27,776 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:54:28,046 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:28,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:29,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:29,777 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:54:30,588 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:54:30,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:31,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:32,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:33,685 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:34,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:35,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:54:35,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:54:35,594 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:54:35,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:36,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:37,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:38,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:39,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:40,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:40,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:54:41,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:54:41,275 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:54:41,275 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:54:41,277 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:54:41,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:41,787 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:54:41,787 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:54:42,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:43,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:44,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:45,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:46,092 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:54:46,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:54:46,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:47,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:48,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:49,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:50,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:54:50,594 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:54:50,688 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:51,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:51,747 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:54:52,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:53,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:54,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:55,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:54:55,093 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:54:55,094 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:54:55,096 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:54:55,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:55,798 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:54:55,798 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:54:56,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:57,134 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:54:57,690 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:58,691 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:54:59,691 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:00,691 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:01,691 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:02,135 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:55:02,691 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:03,691 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:04,692 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:05,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:55:05,594 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:55:05,692 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:06,692 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:07,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:55:07,692 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:08,692 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:08,868 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:55:08,870 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:55:08,870 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:55:08,872 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:55:09,692 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:09,809 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:55:09,809 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:55:10,692 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:11,692 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:12,692 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:12,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:55:13,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:14,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:15,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:16,095 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:55:16,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:17,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:18,098 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:55:18,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:19,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:20,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:55:20,594 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:55:20,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:21,694 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:22,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:55:22,634 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:55:22,635 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:55:22,636 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:55:22,818 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:55:22,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:23,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:55:23,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:23,819 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:55:24,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:25,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:26,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:27,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:28,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:55:28,796 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:29,796 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:30,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:31,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:32,796 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:33,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:55:33,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:34,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:35,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:55:35,594 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:55:35,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:36,465 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:55:36,466 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:55:36,467 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:55:36,469 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:55:36,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:36,828 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:55:37,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:37,829 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:55:38,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:39,507 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:55:39,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:40,797 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:41,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:42,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:43,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:44,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:55:44,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:45,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:46,099 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:55:46,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:47,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:48,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:49,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:50,082 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:55:50,084 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:55:50,084 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:55:50,085 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:55:50,086 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:55:50,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:55:50,595 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:55:50,798 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:50,839 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:55:51,799 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:51,840 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:55:52,799 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:53,799 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:54,799 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:55,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:55:55,799 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:56,799 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:57,799 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:58,799 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:55:59,799 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:00,709 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:56:00,800 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:01,800 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:02,800 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:03,800 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:03,872 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:56:03,874 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:56:03,875 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:56:03,876 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:56:04,800 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:04,849 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:56:05,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:56:05,595 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:56:05,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:56:05,800 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:05,850 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:56:06,800 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:07,800 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:08,926 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:09,925 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:10,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:56:10,925 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:11,925 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:12,926 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:13,926 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:14,926 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:15,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:56:15,926 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:16,103 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:56:16,926 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:17,545 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:56:17,547 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:56:17,547 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:56:17,549 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:56:17,859 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:56:17,860 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:56:17,926 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:18,926 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:19,926 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:20,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:56:20,595 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:56:20,926 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:21,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:56:21,927 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:22,927 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:23,927 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:24,927 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:25,927 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:26,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:56:26,927 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:27,927 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:28,927 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:29,928 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:30,928 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:31,409 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:56:31,410 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:56:31,411 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:56:31,413 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:56:31,870 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:56:31,870 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:56:31,928 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:32,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:56:32,928 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:33,928 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:34,928 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:35,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:56:35,595 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:56:35,928 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:36,929 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:37,692 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:56:37,929 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:38,929 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:39,929 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:40,929 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:41,929 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:42,693 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:56:42,929 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:43,929 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:44,929 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:45,099 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:56:45,100 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:56:45,101 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:56:45,102 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:56:45,881 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:56:45,881 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:56:45,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:46,106 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:56:46,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:47,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:48,109 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:56:48,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:49,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:50,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:56:50,595 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:56:50,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:51,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:52,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:53,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:56:53,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:54,931 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:55,931 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:56,931 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:57,931 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:58,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:56:58,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:56:58,850 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:56:58,850 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:56:58,852 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:56:58,891 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:56:59,093 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:56:59,892 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:57:00,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:01,165 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:02,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:03,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:03,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:57:04,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:05,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:05,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:57:05,595 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:57:06,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:07,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:08,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:09,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:09,703 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:57:10,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:11,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:12,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:12,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:57:12,497 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:57:12,497 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:57:12,499 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:57:12,902 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:57:13,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:13,903 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:57:14,164 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:15,165 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:15,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:57:16,110 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:57:16,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:17,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:18,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:19,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:20,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:20,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:57:20,595 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:57:20,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:57:21,178 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:22,179 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:23,179 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:24,179 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:25,179 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:25,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:57:26,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:57:26,156 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:57:26,157 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:57:26,158 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:57:26,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:26,912 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:57:27,395 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:27,913 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:57:28,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:29,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:30,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:31,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:57:31,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:32,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:33,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:34,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:35,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:35,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:57:35,595 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:57:36,482 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:36,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:57:37,482 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:38,482 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:39,482 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:39,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:57:39,882 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:57:39,883 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:57:39,885 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:57:39,922 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:57:40,482 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:41,482 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:41,924 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:57:41,925 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:57:42,482 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:43,576 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:44,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:45,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:46,114 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:57:46,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:47,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:57:47,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:48,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:49,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:50,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:50,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:57:50,595 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:57:51,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:52,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:52,713 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:57:53,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:53,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:57:53,573 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:57:53,574 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:57:53,576 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:57:53,935 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:57:54,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:54,936 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:57:55,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:56,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:57,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:58,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:57:58,614 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:57:59,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:00,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:01,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:02,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:03,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:03,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:58:04,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:05,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:05,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:58:05,595 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:58:06,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:07,213 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:58:07,215 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:58:07,215 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:58:07,217 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:58:07,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:07,945 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:58:08,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:08,945 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:58:09,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:58:09,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:10,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:11,565 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:12,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:13,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:14,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:58:14,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:15,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:16,118 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:58:16,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:17,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:18,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:19,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:20,120 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:58:20,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:20,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:58:20,596 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:58:20,975 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:58:20,977 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:58:20,977 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:58:20,979 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:58:21,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:21,956 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:58:22,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:22,957 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:58:23,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:24,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:25,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:25,980 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:58:26,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:27,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:28,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:29,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:30,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:30,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:58:31,665 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:32,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:33,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:34,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:34,747 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:58:34,749 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:58:34,750 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:58:34,751 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:58:34,966 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:58:35,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:58:35,596 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:58:35,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:36,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:36,683 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:58:36,968 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:58:37,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:38,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:39,785 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:40,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:41,684 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:58:41,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:42,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:43,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:44,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:45,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:46,121 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:58:46,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:47,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:58:47,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:48,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:58:48,418 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:58:48,418 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:58:48,420 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:58:48,772 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:48,976 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:58:49,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:50,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:58:50,734 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:58:50,775 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:50,978 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:58:51,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:52,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:52,803 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:58:53,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:54,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:55,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:56,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:57,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:57,804 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:58:58,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:58:59,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:00,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:01,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:02,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:59:02,197 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:59:02,197 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:59:02,200 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:59:02,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:02,987 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:59:03,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:59:03,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:04,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:04,988 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:59:05,722 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:59:05,722 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:59:05,897 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:06,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:07,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:08,858 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:59:08,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:09,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:10,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:11,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:12,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:13,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:59:13,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:14,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:15,795 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:59:15,797 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:59:15,798 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:59:15,799 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:59:15,997 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:59:16,110 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:16,126 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:59:16,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:17,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:18,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:18,999 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:59:19,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:59:19,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:20,722 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:59:20,723 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:59:20,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:21,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:22,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:23,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:24,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:59:24,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:25,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:26,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:27,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:28,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:29,346 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:59:29,348 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:59:29,348 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:59:29,350 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:59:29,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:30,007 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:59:30,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:59:30,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:31,008 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:59:31,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:33,031 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:34,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:35,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:35,388 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:59:35,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:59:35,723 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:59:36,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:37,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:38,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:39,028 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:40,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:40,866 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:59:41,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:42,029 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:42,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:59:42,837 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:59:42,838 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:59:42,839 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:59:43,016 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:59:43,069 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:44,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:45,018 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:59:45,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:45,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:59:46,128 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-04 23:59:46,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:47,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:48,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:49,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:50,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:50,722 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 23:59:50,723 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-04 23:59:51,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:51,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:59:52,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:53,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:54,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:55,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:56,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:56,338 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 23:59:56,339 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-04 23:59:56,340 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-04 23:59:56,341 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 23:59:57,026 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-04 23:59:57,157 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:57,378 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 23:59:58,157 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 23:59:59,028 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-04 23:59:59,157 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:00,157 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:01,157 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:02,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:02,379 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:00:03,157 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:04,157 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:05,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:05,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:00:05,735 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:00:06,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:07,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:07,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:00:08,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:09,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:09,851 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:00:09,852 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:00:09,853 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:00:09,855 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:00:10,036 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:00:10,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:11,037 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:00:11,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:12,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:12,893 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:00:13,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:14,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:15,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:16,131 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:00:16,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:17,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:18,134 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:00:18,167 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:19,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:20,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:20,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:00:20,723 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:00:21,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:22,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:23,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:23,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:00:23,307 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:00:23,308 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:00:23,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:00:23,309 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:00:24,046 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:00:24,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:25,046 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:00:25,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:26,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:27,169 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:28,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:28,347 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:00:29,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:30,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:31,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:32,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:33,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:33,348 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:00:34,302 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:35,303 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:35,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:00:35,723 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:00:36,303 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:36,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:00:36,754 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:00:36,754 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:00:36,756 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:00:37,055 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:00:37,303 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:38,303 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:38,794 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:00:39,056 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:00:39,303 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:40,379 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:41,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:42,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:43,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:43,795 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:00:44,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:45,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:46,135 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:00:46,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:47,378 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:48,378 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:49,138 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:00:49,378 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:50,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:00:50,188 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:00:50,189 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:00:50,190 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:00:50,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:50,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:00:50,723 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:00:51,064 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:00:51,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:52,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:53,066 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:00:53,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:54,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:54,831 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:00:55,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:56,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:57,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:58,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:59,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:00:59,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:01:00,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:01,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:02,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:03,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:03,588 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:01:03,590 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:01:03,591 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:01:03,592 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:01:04,073 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:01:04,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:05,074 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:01:05,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:05,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:01:05,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:01:05,723 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:01:06,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:07,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:08,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:09,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:10,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:10,799 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:01:11,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:12,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:13,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:14,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:15,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:15,800 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:01:16,139 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:01:16,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:17,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:01:17,020 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:01:17,020 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:01:17,022 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:01:17,083 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:01:17,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:18,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:19,084 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:01:19,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:20,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:20,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:01:20,724 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:01:20,869 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:01:21,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:22,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:23,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:24,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:25,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:25,870 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:01:26,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:27,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:28,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:29,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:30,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:30,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:01:30,515 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:01:30,516 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:01:30,517 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:01:31,091 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:01:31,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:31,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:01:32,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:33,093 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:01:33,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:34,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:35,612 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:35,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:01:35,724 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:01:36,597 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:36,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:01:37,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:38,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:39,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:40,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:41,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:41,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:01:42,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:43,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:43,990 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:01:43,992 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:01:43,992 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:01:43,994 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:01:44,101 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:01:44,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:45,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:46,143 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:01:46,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:47,103 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:01:47,145 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:01:47,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:48,670 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:49,670 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:50,670 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:50,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:01:50,724 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:01:51,670 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:52,671 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:52,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:01:53,671 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:54,671 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:55,671 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:56,671 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:57,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:01:57,499 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:01:57,499 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:01:57,501 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:01:58,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:58,111 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:01:58,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:01:59,010 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:01:59,112 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:02:00,010 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:01,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:02,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:03,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:03,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:02:04,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:05,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:05,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:02:05,724 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:02:06,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:07,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:08,011 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:08,803 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:02:09,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:10,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:11,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:11,118 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:02:11,120 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:02:11,121 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:02:11,122 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:02:12,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:12,122 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:02:13,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:13,123 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:02:14,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:14,160 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:02:15,012 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:16,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:16,144 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:02:17,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:18,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:19,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:20,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:20,146 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:02:20,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:02:20,724 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:02:21,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:22,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:23,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:24,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:24,618 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:02:24,620 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:02:24,621 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:02:24,626 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:02:25,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:25,132 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:02:25,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:02:26,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:27,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:27,133 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:02:28,014 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:29,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:30,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:30,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:02:31,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:32,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:33,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:34,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:35,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:35,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:02:35,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:02:35,724 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:02:36,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:37,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:38,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:38,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:02:38,118 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:02:38,118 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:02:38,120 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:02:38,141 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:02:39,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:40,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:41,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:41,144 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:02:41,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:02:42,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:43,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:44,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:45,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:46,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:46,148 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:02:47,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:47,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:02:48,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:49,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:50,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:50,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:02:50,725 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:02:51,018 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:51,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:02:51,649 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:02:51,649 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:02:51,651 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:02:52,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:52,155 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:02:52,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:02:53,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:53,156 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:02:54,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:55,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:56,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:57,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:57,689 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:02:58,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:02:59,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:00,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:01,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:02,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:02,690 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:03:03,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:04,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:05,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:05,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:03:05,197 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:03:05,198 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:03:05,203 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:03:05,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:03:05,725 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:03:06,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:06,165 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:03:07,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:07,166 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:03:07,812 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:03:08,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:09,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:10,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:11,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:12,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:12,812 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:03:13,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:14,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:15,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:16,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:16,152 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:03:17,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:18,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:18,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:03:18,759 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:03:18,761 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:03:18,761 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:03:18,763 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:03:19,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:19,174 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:03:20,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:20,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:03:20,725 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:03:21,078 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:21,176 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:03:22,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:23,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:23,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:03:24,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:25,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:26,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:27,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:28,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:28,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:03:29,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:30,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:31,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:32,066 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:32,320 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:03:32,322 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:03:32,322 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:03:32,324 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:03:33,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:33,184 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:03:34,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:34,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:03:35,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:35,186 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:03:35,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:03:35,725 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:03:36,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:37,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:38,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:39,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:39,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:03:40,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:41,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:42,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:43,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:44,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:44,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:03:45,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:45,868 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:03:45,870 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:03:45,870 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:03:45,875 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:03:46,155 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:03:46,157 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:46,193 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:03:47,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:47,194 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:03:48,214 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:49,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:50,157 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:03:50,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:50,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:03:50,725 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:03:51,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:52,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:53,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:54,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:55,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:55,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:03:56,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:57,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:58,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:59,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:03:59,420 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:03:59,422 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:03:59,422 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:03:59,423 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:04:00,203 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:04:00,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:01,204 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:04:01,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:01,462 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:04:02,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:03,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:04,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:05,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:05,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:04:05,725 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:04:06,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:06,789 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:04:07,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:08,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:09,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:10,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:11,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:11,789 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:04:12,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:12,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:04:12,850 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:04:12,850 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:04:12,851 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:04:13,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:13,213 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:04:14,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:15,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:15,214 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:04:16,159 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:04:16,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:17,161 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:04:17,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:18,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:19,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:20,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:20,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:04:20,725 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:04:21,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:22,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:22,854 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:04:23,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:24,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:25,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:26,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:26,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:04:26,258 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:04:26,258 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:04:26,260 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:04:27,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:27,222 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:04:28,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:28,298 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:04:29,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:29,224 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:04:30,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:31,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:32,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:33,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:04:33,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:34,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:35,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:35,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:04:35,725 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:04:36,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:37,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:38,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:38,834 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:04:39,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:39,748 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:04:39,749 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:04:39,750 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:04:39,752 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:04:40,232 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:04:40,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:41,233 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:04:41,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:42,315 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:43,377 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:44,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:44,790 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:04:45,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:46,163 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:04:46,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:47,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:48,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:49,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:50,165 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:04:50,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:50,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:04:50,725 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:04:51,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:52,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:53,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:04:53,207 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:04:53,207 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:04:53,209 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:04:53,242 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:04:53,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:54,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:55,244 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:04:55,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:04:55,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:56,527 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:57,527 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:58,527 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:04:59,527 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:00,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:05:00,527 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:01,527 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:02,527 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:03,527 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:04,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:05,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:05:05,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:05,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:05:05,726 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:05:06,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:06,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:05:06,746 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:05:06,746 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:05:06,749 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:05:07,253 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:05:07,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:08,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:09,255 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:05:09,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:10,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:10,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:05:11,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:12,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:13,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:14,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:15,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:15,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:05:16,166 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:05:16,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:17,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:18,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:19,529 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:20,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:05:20,199 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:05:20,200 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:05:20,201 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:05:20,266 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:05:20,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:20,728 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:05:20,729 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:05:20,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:05:21,267 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:05:21,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:22,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:23,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:24,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:25,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:25,811 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:05:26,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:27,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:28,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:29,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:30,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:30,811 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:05:31,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:32,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:33,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:33,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:05:33,634 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:05:33,634 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:05:33,636 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:05:34,277 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:05:34,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:35,278 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:05:35,531 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:35,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:05:35,726 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:05:35,868 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:05:36,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:37,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:38,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:39,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:40,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:40,869 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:05:41,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:42,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:43,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:44,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:45,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:45,869 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:05:46,170 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:05:46,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:47,125 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:05:47,127 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:05:47,127 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:05:47,129 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:05:47,287 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:05:47,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:48,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:49,288 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:05:49,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:50,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:50,747 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:05:50,748 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:05:51,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:51,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:05:52,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:53,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:54,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:55,534 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:56,534 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:56,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:05:57,534 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:58,534 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:05:59,534 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:00,534 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:00,638 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:06:00,640 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:06:00,640 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:06:00,642 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:06:01,298 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:06:01,534 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:02,534 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:02,680 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:06:03,300 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:06:03,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:04,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:05,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:05,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:06:05,730 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:06:06,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:07,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:07,829 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:06:08,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:09,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:10,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:11,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:12,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:12,830 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:06:13,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:14,264 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:06:14,266 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:06:14,266 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:06:14,268 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:06:14,308 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:06:14,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:15,309 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:06:15,536 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:16,173 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:06:16,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:17,592 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:18,176 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:06:18,592 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:19,592 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:20,592 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:20,729 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:06:20,730 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:06:21,592 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:22,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:23,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:23,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:06:24,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:25,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:26,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:27,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:27,729 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:06:27,730 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:06:27,731 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:06:27,732 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:06:28,319 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:06:28,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:29,320 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:06:29,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:29,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:06:30,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:31,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:32,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:33,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:34,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:34,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:06:35,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:35,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:06:35,730 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:06:36,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:37,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:38,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:39,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:39,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:06:40,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:41,184 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:06:41,185 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:06:41,186 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:06:41,187 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:06:41,330 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:06:41,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:42,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:43,332 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:06:43,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:44,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:45,226 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:06:45,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:46,177 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:06:46,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:47,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:48,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:49,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:50,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:50,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:06:50,730 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:06:50,828 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:06:51,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:52,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:53,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:54,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:54,664 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:06:54,665 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:06:54,666 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:06:54,667 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:06:55,341 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:06:55,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:56,597 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:56,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:06:57,343 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:06:57,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:58,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:06:59,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:00,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:01,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:01,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:07:02,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:03,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:04,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:05,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:05,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:07:05,730 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:07:06,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:06,843 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:07:07,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:08,195 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:07:08,196 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:07:08,197 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:07:08,199 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:07:08,351 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:07:08,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:09,352 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:07:09,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:10,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:11,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:12,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:07:12,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:13,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:14,599 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:15,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:16,182 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:07:16,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:17,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:18,184 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:07:18,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:19,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:20,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:20,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:07:20,731 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:07:21,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:21,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:07:21,703 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:07:21,704 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:07:21,706 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:07:22,361 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:07:22,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:23,363 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:07:23,600 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:23,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:07:24,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:25,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:26,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:27,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:28,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:28,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:07:29,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:30,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:31,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:32,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:33,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:33,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:07:34,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:35,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:07:35,300 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:07:35,301 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:07:35,303 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:07:35,372 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:07:35,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:35,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:07:35,731 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:07:36,602 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:37,374 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:07:37,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:38,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:38,898 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:07:39,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:40,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:41,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:42,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:43,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:43,898 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:07:44,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:45,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:46,185 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:07:46,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:47,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:48,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:48,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:07:48,885 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:07:48,886 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:07:48,887 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:07:48,924 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:07:49,382 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:07:49,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:50,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:50,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:07:50,731 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:07:51,384 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:07:51,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:52,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:53,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:54,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:54,833 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:07:55,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:56,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:57,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:58,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:59,618 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:07:59,834 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:08:00,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:01,618 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:02,402 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:08:02,404 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:08:02,404 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:08:02,406 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:08:02,648 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:03,394 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:08:03,394 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:08:03,645 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:04,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:05,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:08:05,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:08:05,731 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:08:05,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:06,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:07,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:08,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:09,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:10,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:10,812 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:08:11,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:12,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:13,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:14,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:15,737 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:15,813 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:08:15,902 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:08:15,904 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:08:15,904 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:08:15,906 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:08:16,188 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:08:16,403 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:08:16,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:17,404 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:08:17,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:18,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:19,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:20,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:08:20,731 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:08:20,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:21,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:21,800 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:08:22,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:23,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:24,738 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:25,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:26,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:26,800 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:08:27,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:28,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:29,469 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:08:29,471 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:08:29,471 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:08:29,473 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:08:29,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:30,414 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:08:30,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:31,414 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:08:31,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:32,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:08:32,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:33,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:34,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:35,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:08:35,732 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:08:35,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:36,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:37,787 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:37,811 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:08:38,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:39,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:40,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:41,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:42,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:42,812 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:08:43,088 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:08:43,090 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:08:43,091 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:08:43,092 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:08:43,423 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:08:43,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:44,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:45,425 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:08:45,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:46,192 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:08:46,791 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:47,791 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:48,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:08:48,791 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:49,791 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:50,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:08:50,731 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:08:50,791 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:51,791 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:52,791 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:53,792 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:53,821 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:08:54,792 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:55,792 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:56,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:08:56,621 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:08:56,621 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:08:56,623 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:08:56,866 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:57,434 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:08:57,435 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:08:57,861 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:58,951 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:08:59,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:08:59,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:00,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:01,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:02,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:03,938 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:04,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:09:04,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:05,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:09:05,732 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:09:05,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:06,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:07,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:08,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:09,821 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:09:09,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:10,118 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:09:10,119 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:09:10,120 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:09:10,122 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:09:10,444 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:09:10,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:11,445 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:09:11,939 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:12,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:13,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:14,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:15,159 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:09:15,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:16,195 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:09:16,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:17,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:18,940 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:19,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:20,198 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:09:20,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:09:20,732 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:09:20,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:21,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:22,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:23,782 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:09:23,783 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:09:23,784 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:09:23,786 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:09:24,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:24,455 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:09:25,013 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:25,456 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:09:25,823 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:09:26,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:27,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:28,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:29,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:30,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:30,824 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:09:31,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:32,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:33,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:34,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:35,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:35,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:09:35,732 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:09:35,877 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:09:36,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:37,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:37,410 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:09:37,411 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:09:37,412 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:09:37,413 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:09:37,466 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:09:38,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:39,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:39,468 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:09:40,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:41,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:41,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:09:42,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:43,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:44,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:45,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:46,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:46,199 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:09:47,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:47,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:09:48,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:49,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:50,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:50,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:09:50,743 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:09:50,979 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:09:50,981 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:09:50,981 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:09:50,983 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:09:51,225 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:51,477 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:09:51,477 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:09:52,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:52,984 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:09:53,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:54,122 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:55,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:56,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:57,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:57,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:09:58,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:09:59,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:00,123 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:01,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:02,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:02,986 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:10:03,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:04,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:04,668 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:10:04,669 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:10:04,670 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:10:04,671 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:10:05,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:05,488 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:10:05,488 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:10:05,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:10:05,733 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:10:06,125 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:07,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:08,125 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:08,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:10:09,125 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:10,125 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:11,125 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:12,125 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:13,125 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:13,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:10:14,126 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:15,126 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:16,126 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:16,203 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:10:17,126 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:18,126 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:18,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:10:18,260 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:10:18,260 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:10:18,262 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:10:18,498 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:10:19,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:19,300 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:10:19,499 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:10:20,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:20,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:10:20,733 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:10:21,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:22,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:23,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:24,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:24,830 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:10:25,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:26,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:27,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:28,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:29,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:29,831 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:10:30,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:31,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:31,638 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:10:31,640 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:10:31,640 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:10:31,642 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:10:32,129 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:32,509 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:10:33,129 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:33,510 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:10:34,129 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:35,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:35,680 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:10:35,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:10:35,733 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:10:36,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:37,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:38,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:39,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:40,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:40,892 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:10:41,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:42,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:43,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:44,261 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:44,954 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:10:44,956 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:10:44,956 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:10:44,958 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:10:45,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:45,519 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:10:45,520 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:10:45,996 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:10:46,206 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:10:46,262 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:47,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:48,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:49,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:50,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:50,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:10:50,733 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:10:51,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:51,894 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:10:52,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:53,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:54,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:55,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:56,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:56,895 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:10:57,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:58,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:58,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:10:58,535 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:10:58,536 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:10:58,537 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:10:59,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:10:59,529 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:10:59,530 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:11:00,268 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:01,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:02,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:02,575 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:11:03,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:04,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:05,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:05,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:11:05,733 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:11:06,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:07,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:07,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:11:08,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:09,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:10,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:11,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:12,039 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:11:12,040 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:11:12,041 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:11:12,042 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:11:12,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:12,539 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:11:13,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:11:13,277 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:13,540 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:11:14,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:15,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:16,210 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:11:16,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:17,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:18,212 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:11:18,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:19,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:20,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:20,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:11:20,733 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:11:21,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:22,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:23,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:23,857 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:11:24,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:25,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:25,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:11:25,744 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:11:25,744 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:11:25,746 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:11:26,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:26,549 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:11:27,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:27,551 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:11:28,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:29,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:29,784 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:11:30,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:31,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:32,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:33,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:34,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:34,785 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:11:35,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:35,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:11:35,734 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:11:36,480 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:37,480 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:38,480 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:39,289 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:11:39,291 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:11:39,291 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:11:39,293 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:11:39,535 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:39,563 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:11:39,564 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:11:40,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:11:40,532 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:41,608 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:42,594 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:43,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:44,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:45,331 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:11:45,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:46,214 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:11:46,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:47,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:48,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:49,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:50,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:50,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:11:50,734 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:11:50,898 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:11:51,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:52,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:52,856 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:11:52,858 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:11:52,858 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:11:52,860 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:11:53,575 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:11:53,575 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:11:53,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:54,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:55,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:55,899 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:11:56,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:57,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:58,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:11:59,596 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:00,597 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:00,899 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:12:01,597 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:02,597 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:03,597 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:04,597 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:05,597 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:05,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:12:05,734 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:12:06,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:12:06,440 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:12:06,441 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:12:06,441 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:12:06,442 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:12:06,586 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:12:06,677 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:07,587 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:12:07,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:08,755 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:09,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:10,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:11,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:12:11,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:12,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:13,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:14,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:15,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:16,218 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:12:16,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:17,220 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:12:17,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:18,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:19,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:19,913 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:12:19,914 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:12:19,915 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:12:19,916 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:12:20,596 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:12:20,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:12:20,734 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:12:20,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:21,597 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:12:21,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:22,741 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:22,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:12:23,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:24,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:25,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:26,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:27,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:27,842 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:12:28,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:29,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:30,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:31,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:32,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:32,843 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:12:33,402 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:12:33,403 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:12:33,404 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:12:33,405 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:12:33,606 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:12:33,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:34,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:35,608 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:12:35,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:12:35,734 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:12:35,774 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:36,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:37,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:38,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:38,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:12:39,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:40,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:41,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:42,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:43,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:43,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:12:44,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:45,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:46,222 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:12:46,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:46,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:12:46,753 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:12:46,754 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:12:46,755 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:12:47,617 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:12:47,618 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:12:47,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:48,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:49,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:49,794 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:12:50,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:12:50,734 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:12:50,774 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:51,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:52,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:53,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:54,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:54,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:12:55,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:56,745 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:57,746 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:58,746 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:59,746 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:12:59,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:13:00,746 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:01,746 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:02,746 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:03,746 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:04,746 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:04,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:13:05,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:13:05,734 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:13:05,775 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:06,747 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:07,747 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:08,747 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:09,747 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:10,747 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:10,821 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:13:11,747 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:12,894 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:13,635 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:13:14,592 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:15,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:13:16,225 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:13:21,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:13:21,432 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:13:22,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:23,437 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:13:23,437 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:13:23,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:26,559 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:13:29,369 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:13:30,920 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:31,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:13:36,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:13:38,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:39,214 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:13:39,215 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:13:40,129 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:42,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:13:46,228 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:13:47,445 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:47,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:13:48,231 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:13:48,269 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:53,232 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:13:55,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:13:56,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:56,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:13:56,916 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:13:57,587 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:13:59,058 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:14:04,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:14:04,972 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:14:09,060 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:14:09,444 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:09,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:09,841 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:14:10,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:10,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:14:10,836 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:14:11,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:11,844 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:14:12,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:13,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:14,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:14,905 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:14:15,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:16,232 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:14:16,447 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:17,447 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:18,447 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:19,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:20,234 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:14:20,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:21,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:22,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:23,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:24,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:25,235 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:14:25,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:25,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:14:25,841 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:14:26,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:27,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:28,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:29,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:30,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:30,927 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:14:31,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:32,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:33,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:34,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:35,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:35,928 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:14:36,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:37,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:38,451 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:39,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:40,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:40,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:14:40,840 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:14:40,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:14:41,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:42,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:43,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:44,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:45,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:45,936 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:14:46,235 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:14:46,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:47,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:48,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:49,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:50,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:51,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:14:51,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:52,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:53,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:54,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:55,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:55,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:14:55,840 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:14:56,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:56,928 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:14:57,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:58,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:14:59,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:00,455 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:01,455 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:01,929 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:15:02,455 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:03,455 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:04,455 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:05,455 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:06,455 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:06,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:15:11,931 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:15:16,239 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:15:17,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:15:22,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:15:27,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:15:28,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:15:28,234 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:15:28,234 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:28,235 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:15:29,215 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:30,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:31,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:32,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:32,342 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:15:33,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:34,199 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:35,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:36,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:37,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:37,343 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:15:38,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:39,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:40,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:41,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:42,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:42,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:15:43,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:15:43,188 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:15:43,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:44,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:45,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:46,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:46,243 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:15:47,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:48,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:48,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:15:49,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:50,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:51,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:52,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:53,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:53,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:15:54,201 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:55,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:56,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:57,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:58,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:15:58,188 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:15:58,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:15:58,290 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:15:59,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:00,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:01,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:02,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:03,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:03,290 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:16:04,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:05,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:06,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:07,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:08,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:08,291 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:16:09,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:10,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:11,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:12,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:13,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:16:13,188 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:16:13,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:14,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:14,263 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:16:15,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:16,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:16,248 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:16:17,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:18,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:19,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:20,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:20,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:16:21,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:22,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:23,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:24,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:25,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:25,253 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:16:26,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:27,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:28,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:16:28,188 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:16:28,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:29,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:30,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:30,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:16:31,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:32,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:33,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:34,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:35,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:35,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:16:36,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:37,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:38,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:39,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:40,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:40,285 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:16:41,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:42,206 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:43,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:16:43,188 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:16:43,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:44,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:45,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:45,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:16:46,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:46,250 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:16:47,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:48,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:49,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:50,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:51,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:51,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:16:52,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:53,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:54,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:55,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:56,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:56,253 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:16:57,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:58,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:16:58,188 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:16:58,228 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:16:59,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:00,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:01,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:01,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:17:02,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:03,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:04,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:05,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:06,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:06,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:17:07,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:08,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:09,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:10,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:11,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:11,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:17:12,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:13,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:17:13,188 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:17:13,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:14,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:15,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:16,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:16,253 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:17:17,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:17,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:17:18,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:19,210 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:17:22,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:17:27,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:17:32,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:17:37,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:17:42,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:17:46,257 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:17:47,259 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:17:52,260 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:17:57,260 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:18:01,980 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:18:02,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:02,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:18:02,022 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:18:02,980 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:03,112 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:18:03,980 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:04,981 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:05,980 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:06,711 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:18:06,980 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:08,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:08,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:18:09,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:10,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:11,089 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:12,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:13,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:13,475 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:18:14,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:15,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:16,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:16,258 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:18:16,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:18:16,985 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:18:17,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:18,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:18,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:18:18,198 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:18:18,199 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:18:18,200 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:18:18,720 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:18:19,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:19,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:18:20,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:20,722 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:18:21,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:22,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:23,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:24,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:24,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:18:25,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:26,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:27,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:28,091 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:29,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:29,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:18:30,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:31,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:31,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:18:31,555 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:18:31,556 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:18:31,558 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:18:31,731 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:18:31,987 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:18:31,988 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:18:32,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:32,732 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:18:33,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:34,092 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:35,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:18:35,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:36,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:37,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:38,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:39,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:40,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:18:40,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:41,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:42,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:43,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:44,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:44,785 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:18:44,787 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:18:44,788 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:18:44,789 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:18:45,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:45,742 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:18:45,827 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:18:46,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:46,261 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:18:46,743 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:18:46,988 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:18:46,988 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:18:47,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:48,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:49,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:50,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:51,127 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:18:51,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:52,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:53,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:54,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:55,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:56,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:18:56,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:57,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:58,192 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:18:58,195 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:18:58,195 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:18:58,197 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:18:58,435 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:18:58,751 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:18:59,435 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:00,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:00,753 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:19:01,234 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:19:01,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:01,988 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:19:01,988 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:19:02,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:03,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:04,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:05,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:06,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:07,104 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:19:07,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:08,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:09,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:10,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:11,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:11,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:19:11,575 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:19:11,575 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:19:11,577 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:19:11,761 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:19:12,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:12,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:19:12,762 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:19:13,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:14,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:15,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:16,264 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:19:16,530 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:16,988 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:19:16,989 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:19:17,513 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:18,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:19:18,513 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:19,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:20,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:21,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:22,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:23,108 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:19:23,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:24,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:24,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:19:24,725 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:19:24,726 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:19:24,727 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:19:24,771 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:19:25,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:26,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:26,772 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:19:27,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:28,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:28,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:19:29,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:30,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:31,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:31,988 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:19:31,989 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:19:32,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:33,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:34,130 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:19:34,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:35,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:36,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:37,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:38,090 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:19:38,091 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:19:38,092 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:19:38,093 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:19:38,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:38,780 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:19:39,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:19:39,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:40,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:40,781 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:19:41,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:42,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:43,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:44,131 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:19:44,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:45,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:46,268 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:19:46,516 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:46,988 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:19:46,989 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:19:47,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:48,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:49,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:50,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:19:50,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:51,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:19:51,357 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:19:51,358 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:19:51,359 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:19:51,592 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:51,790 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:19:52,589 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:52,791 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:19:53,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:54,652 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:55,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:19:55,652 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:56,652 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:57,652 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:58,652 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:19:59,652 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:00,398 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:20:00,652 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:01,653 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:01,988 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:20:01,989 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:20:02,653 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:03,653 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:04,610 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:20:04,611 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:20:04,612 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:20:04,622 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:20:04,810 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:20:04,847 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:05,651 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:20:05,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:06,742 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:06,814 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:20:07,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:08,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:09,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:10,651 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:20:10,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:11,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:12,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:13,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:14,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:15,652 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:20:15,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:16,272 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:20:16,743 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:16,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:20:16,989 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:20:17,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:18,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:20:18,152 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:20:18,152 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:20:18,154 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:20:18,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:18,823 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:20:19,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:20,744 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:20,825 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:20:21,192 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:20:21,837 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:22,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:23,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:24,818 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:25,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:26,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:20:26,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:27,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:28,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:29,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:30,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:31,194 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:20:31,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:20:31,529 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:20:31,530 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:20:31,531 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:20:31,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:31,833 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:20:31,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:20:31,989 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:20:32,819 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:32,834 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:20:33,844 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:34,834 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:35,834 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:36,834 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:37,147 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:20:37,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:38,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:39,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:40,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:41,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:42,149 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:20:42,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:43,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:44,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:44,921 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:20:44,923 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:20:44,923 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:20:44,925 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:20:45,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:45,843 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:20:46,275 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:20:46,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:46,844 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:20:46,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:20:46,989 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:20:47,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:48,060 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:20:48,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:49,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:50,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:51,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:52,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:53,061 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:20:53,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:54,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:55,837 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:56,837 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:57,837 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:58,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:20:58,471 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:20:58,473 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:20:58,473 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:20:58,474 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:20:58,837 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:20:58,852 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:20:59,837 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:00,837 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:00,854 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:21:01,837 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:01,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:21:01,989 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:21:02,837 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:03,082 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:21:03,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:04,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:05,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:06,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:07,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:08,083 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:21:08,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:09,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:10,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:11,838 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:11,985 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:21:11,986 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:21:11,986 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:21:11,988 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:21:12,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:12,862 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:21:12,863 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:21:13,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:14,025 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:21:14,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:15,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:16,279 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:21:16,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:17,124 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:21:17,124 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:21:17,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:18,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:19,196 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:21:19,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:20,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:21,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:22,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:23,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:24,197 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:21:24,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:25,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:21:25,542 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:21:25,543 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:21:25,544 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:21:25,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:25,872 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:21:26,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:26,873 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:21:27,941 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:28,879 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:29,583 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:21:29,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:30,879 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:31,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:32,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:21:32,106 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:21:32,879 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:33,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:34,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:35,222 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:21:35,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:36,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:37,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:38,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:38,992 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:21:38,993 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:21:38,994 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:21:38,995 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:21:39,880 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:39,883 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:21:40,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:40,885 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:21:41,033 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:21:41,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:42,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:43,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:44,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:45,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:46,034 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:21:46,284 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:21:46,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:47,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:21:47,106 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:21:47,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:48,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:49,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:50,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:51,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:21:51,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:52,463 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:21:52,464 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:21:52,465 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:21:52,467 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:21:52,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:52,894 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:21:53,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:54,882 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:54,895 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:21:55,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:56,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:21:56,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:57,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:58,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:21:59,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:00,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:01,506 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:22:01,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:02,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:22:02,106 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:22:02,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:03,883 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:04,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:05,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:05,918 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:22:05,920 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:22:05,920 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:22:05,922 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:22:06,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:06,904 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:22:06,904 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:22:06,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:22:07,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:08,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:09,884 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:10,952 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:11,951 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:11,960 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:22:12,951 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:13,951 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:14,951 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:15,952 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:16,287 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:22:16,952 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:17,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:22:17,107 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:22:17,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:22:17,952 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:18,952 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:19,263 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:22:19,265 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:22:19,265 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:22:19,266 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:22:19,913 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:22:19,952 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:20,914 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:22:20,952 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:21,952 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:22,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:22:22,953 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:23,953 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:24,953 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:25,953 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:26,953 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:27,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:22:27,953 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:28,953 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:29,953 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:30,953 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:31,954 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:32,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:22:32,106 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:22:32,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:22:32,629 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:22:32,630 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:22:32,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:22:32,631 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:22:32,923 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:22:32,954 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:33,954 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:34,924 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:22:34,959 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:35,956 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:36,956 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:37,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:22:37,956 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:38,956 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:39,956 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:40,956 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:41,957 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:42,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:22:42,957 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:43,957 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:44,957 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:45,957 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:45,965 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:22:45,967 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:22:45,967 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:22:45,968 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:22:46,290 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:22:46,934 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:22:46,934 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:22:46,957 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:47,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:22:47,106 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:22:47,957 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:48,202 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:22:48,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:49,958 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:51,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:52,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:53,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:53,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:22:54,067 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:55,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:56,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:57,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:58,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:58,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:22:59,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:22:59,380 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:22:59,382 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:22:59,383 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:22:59,384 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:22:59,943 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:23:00,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:00,945 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:23:01,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:02,069 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:02,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:23:02,107 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:23:03,073 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:03,264 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:23:04,070 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:05,070 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:06,070 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:07,070 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:08,070 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:08,265 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:23:09,070 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:10,071 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:11,071 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:12,071 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:12,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:23:12,837 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:23:12,838 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:23:12,839 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:23:12,953 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:23:13,077 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:13,876 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:23:14,074 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:14,954 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:23:15,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:16,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:16,294 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:23:17,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:23:17,107 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:23:17,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:18,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:19,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:19,192 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:23:20,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:21,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:22,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:23,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:24,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:24,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:23:25,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:26,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:26,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:23:26,241 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:23:26,242 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:23:26,243 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:23:26,963 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:23:27,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:28,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:28,965 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:23:29,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:29,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:23:30,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:31,299 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:32,106 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:23:32,107 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:23:32,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:33,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:34,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:35,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:35,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:23:36,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:37,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:38,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:39,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:39,569 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:23:39,571 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:23:39,571 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:23:39,573 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:23:39,973 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:23:40,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:40,611 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:23:40,974 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:23:41,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:42,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:43,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:44,229 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:45,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:45,612 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:23:46,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:46,297 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:23:47,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:23:47,107 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:23:47,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:48,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:49,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:50,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:51,223 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:23:51,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:52,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:52,996 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:23:52,998 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:23:52,998 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:23:53,000 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:23:53,236 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:53,983 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:23:54,234 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:54,984 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:23:55,305 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:56,292 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:57,038 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:23:57,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:58,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:23:59,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:00,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:01,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:02,039 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:24:02,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:24:02,107 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:24:02,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:03,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:04,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:05,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:06,293 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:06,480 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:24:06,482 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:24:06,482 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:24:06,484 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:24:06,993 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:24:07,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:07,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:24:08,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:08,995 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:24:09,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:10,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:11,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:12,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:12,523 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:24:13,294 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:14,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:15,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:16,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:16,302 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:24:17,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:24:17,108 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:24:17,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:18,211 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:24:18,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:19,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:19,975 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:24:19,976 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:24:19,977 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:24:19,978 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:24:20,002 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:24:20,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:21,003 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:24:21,295 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:22,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:23,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:24,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:24:24,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:25,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:26,306 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:27,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:28,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:29,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:24:29,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:30,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:31,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:32,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:24:32,108 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:24:32,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:33,307 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:33,479 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:24:33,481 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:24:33,481 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:24:33,482 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:24:34,011 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:24:34,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:34,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:24:35,012 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:24:35,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:36,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:37,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:38,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:39,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:39,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:24:40,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:41,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:42,308 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:43,309 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:44,309 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:44,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:24:45,309 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:46,305 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:24:46,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:46,949 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:24:46,951 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:24:46,952 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:24:46,953 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:24:47,020 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:24:47,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:24:47,183 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:24:47,309 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:48,309 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:49,022 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:24:49,309 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:50,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:24:50,309 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:51,309 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:52,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:53,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:54,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:55,253 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:24:55,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:56,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:57,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:58,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:24:59,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:00,254 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:25:00,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:00,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:25:00,440 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:25:00,441 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:25:00,444 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:25:01,030 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:25:01,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:02,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:25:02,182 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:25:02,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:03,032 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:25:03,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:04,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:05,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:05,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:25:06,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:07,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:08,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:09,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:10,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:10,313 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:25:11,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:12,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:13,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:13,994 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:25:13,996 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:25:13,996 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:25:13,998 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:25:14,040 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:25:14,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:15,041 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:25:15,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:16,036 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:25:16,308 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:25:16,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:17,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:25:17,182 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:25:17,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:18,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:19,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:20,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:21,316 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:25:21,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:22,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:23,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:24,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:25,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:26,317 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:25:26,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:27,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:27,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:25:27,513 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:25:27,514 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:25:27,516 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:25:28,049 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:25:28,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:29,050 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:25:29,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:30,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:31,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:31,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:25:32,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:25:32,182 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:25:32,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:33,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:34,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:35,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:36,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:37,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:25:37,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:38,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:39,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:40,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:41,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:25:41,186 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:25:41,187 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:25:41,188 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:25:41,431 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:42,058 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:25:42,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:43,059 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:25:43,227 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:25:43,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:44,485 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:45,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:46,312 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:25:46,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:47,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:25:47,182 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:25:47,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:48,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:25:48,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:49,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:50,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:51,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:52,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:53,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:25:53,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:54,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:54,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:25:54,639 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:25:54,640 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:25:54,641 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:25:55,067 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:25:55,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:56,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:57,069 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:25:57,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:58,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:25:58,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:25:59,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:00,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:01,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:02,181 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:26:02,182 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:26:02,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:03,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:04,342 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:26:04,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:05,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:06,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:07,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:08,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:26:08,116 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:26:08,117 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:26:08,118 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:26:08,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:09,077 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:26:09,078 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:26:09,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:10,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:26:10,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:11,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:12,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:13,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:14,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:15,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:26:15,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:16,316 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:26:16,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:17,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:26:17,182 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:26:17,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:18,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:19,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:20,314 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:26:20,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:21,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:21,605 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:26:21,607 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:26:21,607 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:26:21,609 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:26:22,088 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:26:22,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:23,089 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:26:23,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:24,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:25,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:25,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:26:26,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:27,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:28,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:29,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:30,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:30,648 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:26:31,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:32,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:26:32,182 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:26:32,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:33,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:34,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:35,078 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:26:35,080 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:26:35,081 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:26:35,082 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:26:35,097 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:26:35,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:36,119 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:26:36,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:37,099 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:26:37,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:38,572 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:39,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:40,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:41,120 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:26:41,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:42,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:43,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:44,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:45,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:46,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:26:46,320 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:26:46,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:47,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:26:47,183 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:26:47,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:48,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:48,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:26:48,663 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:26:48,664 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:26:48,665 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:26:49,106 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:26:49,574 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:50,574 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:51,108 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:26:51,574 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:51,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:26:52,574 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:53,713 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:54,695 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:55,695 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:56,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:56,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:26:57,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:58,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:26:59,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:00,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:01,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:01,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:27:02,121 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:27:02,122 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:27:02,123 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:27:02,124 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:27:02,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:27:02,364 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:27:02,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:03,118 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:27:03,118 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:27:03,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:04,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:05,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:06,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:07,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:27:07,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:08,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:09,806 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:10,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:11,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:12,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:27:12,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:13,808 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:14,807 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:15,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:27:15,575 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:27:15,575 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:27:15,577 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:27:15,851 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:16,127 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:27:16,323 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:27:16,842 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:17,128 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:27:17,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:27:17,363 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:27:17,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:18,434 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:27:18,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:19,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:20,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:21,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:22,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:23,434 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:27:23,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:24,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:25,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:26,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:27,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:28,435 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:27:28,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:29,045 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:27:29,047 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:27:29,048 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:27:29,049 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:27:29,137 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:27:29,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:30,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:31,138 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:27:31,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:32,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:27:32,363 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:27:32,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:33,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:27:33,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:34,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:35,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:36,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:37,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:38,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:27:38,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:39,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:40,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:41,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:42,690 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:27:42,692 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:27:42,692 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:27:42,694 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:27:42,925 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:43,147 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:27:43,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:27:43,925 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:45,002 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:45,149 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:27:45,995 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:46,327 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:27:46,995 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:47,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:27:47,363 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:27:47,995 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:48,995 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:49,525 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:27:49,995 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:50,995 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:51,995 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:52,995 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:53,996 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:54,526 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:27:54,995 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:55,996 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:56,179 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:27:56,180 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:27:56,181 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:27:56,182 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:27:56,996 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:57,157 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:27:57,158 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:27:57,996 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:58,996 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:27:59,996 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:00,220 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:28:00,997 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:01,997 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:02,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:28:02,363 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:28:02,996 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:03,997 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:04,997 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:05,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:28:05,997 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:06,997 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:07,997 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:08,997 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:09,654 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:28:09,656 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:28:09,656 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:28:09,658 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:28:09,997 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:10,166 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:28:10,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:28:10,997 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:11,167 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:28:11,997 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:12,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:13,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:14,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:15,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:28:15,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:16,330 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:28:16,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:17,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:28:17,364 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:28:17,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:18,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:19,998 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:20,999 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:21,524 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:28:21,999 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:22,999 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:23,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:28:23,245 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:28:23,246 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:28:23,247 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:28:23,999 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:24,176 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:28:24,999 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:25,177 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:28:26,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:26,999 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:27,286 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:28:27,999 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:28,999 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:30,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:31,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:32,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:32,287 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:28:32,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:28:32,364 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:28:33,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:34,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:35,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:36,000 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:36,812 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:28:36,814 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:28:36,814 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:28:36,816 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:28:37,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:37,185 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:28:37,854 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:28:38,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:39,125 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:39,187 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:28:40,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:41,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:42,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:42,854 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:28:43,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:44,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:45,114 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:46,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:46,334 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:28:47,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:47,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:28:47,364 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:28:48,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:48,522 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:28:49,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:50,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:50,183 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:28:50,185 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:28:50,186 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:28:50,187 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:28:50,194 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:28:51,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:51,195 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:28:52,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:53,115 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:54,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:54,225 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:28:55,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:56,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:57,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:58,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:59,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:28:59,226 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:29:00,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:01,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:02,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:02,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:29:02,364 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:29:03,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:03,654 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:29:03,656 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:29:03,656 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:29:03,658 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:29:04,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:04,204 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:29:04,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:29:05,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:05,205 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:29:06,117 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:07,117 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:08,117 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:09,117 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:09,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:29:10,117 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:11,118 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:12,117 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:13,117 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:14,118 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:14,697 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:29:15,118 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:16,118 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:16,338 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:29:17,118 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:17,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:29:17,173 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:29:17,174 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:29:17,175 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:29:17,212 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:29:17,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:29:17,417 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:29:18,118 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:19,118 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:19,213 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:29:20,119 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:20,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:29:21,119 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:22,226 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:23,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:24,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:25,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:25,569 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:29:26,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:27,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:28,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:29,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:30,208 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:30,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:29:30,562 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:29:30,563 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:29:30,564 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:29:30,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:29:31,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:31,221 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:29:31,222 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:29:32,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:32,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:29:32,416 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:29:33,209 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:34,255 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:35,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:36,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:36,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:29:37,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:38,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:39,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:40,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:41,252 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:41,569 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:29:42,253 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:43,253 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:44,003 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:29:44,005 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:29:44,005 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:29:44,007 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:29:44,230 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:29:44,253 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:45,231 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:29:45,253 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:46,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:46,341 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:29:47,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:47,344 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:29:47,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:29:47,416 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:29:48,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:49,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:50,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:51,334 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:52,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:52,563 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:29:53,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:54,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:55,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:56,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:57,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:57,564 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:29:57,601 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:29:57,603 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:29:57,604 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:29:57,608 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:29:58,239 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:29:58,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:29:59,240 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:29:59,335 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:00,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:01,342 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:02,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:02,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:30:02,416 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:30:02,575 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:30:03,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:04,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:05,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:06,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:07,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:07,576 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:30:08,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:09,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:10,340 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:11,063 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:30:11,065 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:30:11,066 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:30:11,067 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:30:11,249 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:30:11,341 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:12,341 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:13,105 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:30:13,250 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:30:13,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:14,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:15,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:16,345 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:30:16,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:17,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:17,415 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:30:17,416 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:30:18,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:18,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:30:19,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:20,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:21,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:22,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:23,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:23,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:30:24,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:24,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:30:24,503 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:30:24,504 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:30:24,505 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:30:25,258 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:30:25,259 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:30:25,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:26,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:27,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:28,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:28,543 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:30:29,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:30,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:31,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:32,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:32,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:30:32,417 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:30:33,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:34,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:34,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:30:35,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:36,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:37,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:37,922 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:30:37,924 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:30:37,924 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:30:37,926 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:30:38,267 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:30:38,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:39,268 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:30:39,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:39,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:30:40,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:41,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:42,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:43,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:44,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:44,964 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:30:45,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:46,349 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:30:46,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:47,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:47,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:30:47,417 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:30:48,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:49,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:50,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:50,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:30:51,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:51,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:30:51,501 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:30:51,502 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:30:51,503 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:30:52,278 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:30:52,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:53,279 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:30:53,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:54,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:55,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:55,541 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:30:56,364 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:57,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:58,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:30:59,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:00,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:00,542 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:31:01,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:02,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:02,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:31:02,417 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:31:03,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:04,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:05,107 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:31:05,109 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:31:05,109 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:31:05,111 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:31:05,288 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:31:05,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:06,149 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:31:06,366 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:07,290 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:31:07,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:08,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:09,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:10,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:11,149 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:31:11,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:12,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:13,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:14,423 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:15,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:16,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:31:16,353 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:31:16,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:17,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:31:17,417 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:31:17,457 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:18,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:18,550 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:31:18,552 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:31:18,552 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:31:18,554 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:31:19,302 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:31:19,302 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:31:19,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:20,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:21,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:21,592 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:31:22,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:23,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:24,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:25,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:26,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:26,592 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:31:27,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:28,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:29,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:30,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:31,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:31,593 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:31:32,117 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:31:32,118 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:31:32,119 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:31:32,121 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:31:32,313 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:31:32,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:31:32,417 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:31:32,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:33,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:34,315 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:31:34,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:35,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:36,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:37,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:37,570 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:31:38,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:39,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:40,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:41,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:42,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:42,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:31:43,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:44,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:45,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:45,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:31:45,789 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:31:45,790 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:31:45,792 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:31:46,324 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:31:46,356 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:31:46,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:47,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:31:47,417 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:31:47,457 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:48,325 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:31:48,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:48,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:31:49,547 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:50,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:51,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:52,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:53,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:53,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:31:54,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:55,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:56,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:57,534 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:58,533 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:31:58,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:31:59,416 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:31:59,418 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:31:59,419 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:31:59,420 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:31:59,651 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:00,333 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:32:00,650 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:01,729 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:02,335 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:32:02,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:32:02,417 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:32:02,713 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:03,713 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:04,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:32:04,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:05,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:06,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:07,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:08,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:09,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:32:09,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:10,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:11,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:12,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:12,812 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:32:12,813 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:32:12,814 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:32:12,815 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:32:13,343 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:32:13,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:14,344 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:32:14,714 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:14,853 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:32:15,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:16,361 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:32:16,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:17,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:32:17,417 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:32:17,853 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:18,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:19,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:20,576 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:32:20,835 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:21,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:22,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:23,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:24,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:25,577 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:32:25,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:26,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:32:26,280 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:32:26,280 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:32:26,282 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:32:26,352 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:32:26,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:27,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:28,354 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:32:28,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:29,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:30,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:31,320 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:32:31,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:32,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:32:32,418 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:32:32,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:33,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:34,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:35,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:36,578 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:32:36,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:37,839 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:38,840 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:39,832 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:32:39,833 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:32:39,834 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:32:39,835 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:32:40,068 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:40,363 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:32:40,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:41,873 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:32:41,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:42,365 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:32:42,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:43,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:44,971 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:45,970 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:46,363 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:32:46,971 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:47,365 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:32:47,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:32:47,418 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:32:47,971 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:48,971 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:49,971 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:50,971 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:51,971 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:52,557 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:32:52,971 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:53,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:32:53,334 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:32:53,334 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:32:53,336 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:32:53,375 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:32:53,972 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:54,376 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:32:54,972 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:55,972 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:57,017 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:58,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:32:58,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:32:59,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:00,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:01,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:02,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:02,417 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:33:02,418 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:33:03,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:03,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:33:04,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:05,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:06,016 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:06,881 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:33:06,883 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:33:06,884 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:33:06,885 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:33:07,128 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:07,385 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:33:08,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:08,386 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:33:08,923 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:33:09,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:10,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:11,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:12,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:13,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:13,924 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:33:14,019 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:15,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:16,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:16,367 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:33:17,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:17,418 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:33:17,418 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:33:18,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:19,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:19,534 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:33:20,020 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:20,384 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:33:20,386 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:33:20,387 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:33:20,388 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:33:20,394 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:33:21,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:22,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:22,395 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:33:23,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:24,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:25,083 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:25,426 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:33:26,083 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:27,083 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:28,083 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:29,083 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:30,083 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:30,427 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:33:31,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:32,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:32,418 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:33:32,418 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:33:33,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:33,845 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:33:33,847 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:33:33,847 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:33:33,849 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:33:34,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:34,404 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:33:35,084 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:35,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:33:36,161 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:36,406 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:33:37,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:38,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:39,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:40,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:40,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:33:41,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:42,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:43,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:44,151 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:45,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:45,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:33:46,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:46,371 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:33:47,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:47,349 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:33:47,350 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:33:47,351 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:33:47,353 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:33:47,414 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:33:47,580 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:33:47,580 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:33:48,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:48,415 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:33:49,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:50,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:51,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:51,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:33:52,152 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:53,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:54,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:55,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:56,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:56,687 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:33:57,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:58,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:33:59,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:00,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:00,719 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:34:00,720 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:34:00,721 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:34:00,722 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:34:01,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:01,424 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:34:01,760 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:34:02,153 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:02,426 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:34:02,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:34:02,486 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:34:03,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:04,154 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:05,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:06,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:07,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:07,558 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:34:08,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:09,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:10,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:11,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:12,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:12,559 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:34:13,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:14,155 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:14,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:34:14,189 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:34:14,189 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:34:14,191 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:34:14,435 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:34:15,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:16,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:16,375 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:34:16,437 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:34:17,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:17,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:34:17,486 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:34:17,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:34:18,156 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:19,297 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:20,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:21,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:22,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:22,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:34:23,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:24,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:25,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:26,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:27,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:27,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:34:27,725 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:34:27,726 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:34:27,727 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:34:27,728 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:34:28,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:28,445 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:34:29,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:30,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:30,447 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:34:31,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:32,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:32,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:34:32,487 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:34:33,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:33,597 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:34:34,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:35,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:36,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:37,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:38,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:38,598 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:34:39,279 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:40,280 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:41,193 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:34:41,195 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:34:41,196 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:34:41,197 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:34:41,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:41,455 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:34:42,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:42,456 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:34:43,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:44,236 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:34:44,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:45,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:46,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:46,378 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:34:47,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:47,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:34:47,487 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:34:48,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:49,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:49,653 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:34:50,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:51,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:52,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:53,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:54,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:54,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:34:54,633 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:34:54,634 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:34:54,635 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:34:54,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:34:55,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:55,467 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:34:56,337 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:56,468 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:34:57,338 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:58,354 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:59,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:34:59,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:35:00,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:01,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:02,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:02,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:35:02,487 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:35:03,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:04,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:05,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:05,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:35:06,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:07,354 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:08,044 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:35:08,045 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:35:08,046 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:35:08,047 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:35:08,354 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:08,477 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:35:09,354 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:10,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:10,479 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:35:11,086 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:35:11,355 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:12,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:13,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:14,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:15,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:16,087 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:35:16,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:16,378 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:35:17,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:17,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:35:17,487 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:35:18,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:19,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:20,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:21,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:21,386 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:35:21,388 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:35:21,388 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:35:21,389 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:35:21,390 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:35:21,487 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:35:22,359 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:22,488 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:35:23,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:24,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:25,359 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:26,429 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:35:26,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:27,480 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:28,480 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:29,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:30,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:31,430 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:35:31,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:32,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:32,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:35:32,487 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:35:33,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:34,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:34,867 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:35:34,868 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:35:34,869 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:35:34,870 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:35:35,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:35,497 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:35:36,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:36,498 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:35:36,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:35:37,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:38,539 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:39,520 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:40,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:41,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:41,910 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:35:42,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:43,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:44,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:45,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:46,383 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:35:46,521 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:47,385 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:35:47,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:35:47,488 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:35:47,528 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:48,333 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:35:48,334 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:35:48,335 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:35:48,339 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:35:48,506 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:35:48,573 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:49,570 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:50,508 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:35:50,643 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:51,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:52,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:53,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:35:53,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:54,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:55,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:56,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:57,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:58,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:35:58,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:35:59,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:00,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:01,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:01,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:36:01,889 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:36:01,890 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:36:01,891 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:36:02,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:36:02,488 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:36:02,518 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:36:02,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:03,552 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:36:03,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:04,519 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:36:04,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:05,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:06,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:07,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:08,553 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:36:08,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:09,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:10,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:11,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:12,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:13,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:36:13,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:14,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:15,362 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:36:15,364 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:36:15,364 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:36:15,366 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:36:15,527 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:36:15,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:16,386 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:36:16,528 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:36:16,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:17,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:36:17,488 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:36:17,686 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:18,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:36:18,671 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:19,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:20,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:21,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:22,672 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:23,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:36:23,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:24,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:25,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:26,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:27,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:28,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:36:28,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:28,972 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:36:28,974 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:36:28,974 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:36:28,976 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:36:29,537 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:36:29,673 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:30,539 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:36:30,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:31,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:32,487 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:36:32,488 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:36:32,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:33,674 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:34,586 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:36:34,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:35,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:36,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:37,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:38,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:39,587 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:36:39,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:40,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:41,675 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:42,398 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:36:42,399 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:36:42,400 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:36:42,402 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:36:42,548 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:36:42,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:43,676 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:44,549 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:36:44,713 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:45,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:36:45,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:46,390 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:36:46,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:47,510 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:36:47,510 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:36:47,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:48,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:49,699 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:50,678 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:36:50,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:51,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:52,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:53,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:54,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:55,679 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:36:55,700 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:55,886 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:36:55,888 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:36:55,889 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:36:55,890 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:36:56,558 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:36:56,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:57,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:58,560 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:36:58,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:36:59,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:00,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:00,928 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:37:01,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:02,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:37:02,495 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:37:02,701 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:03,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:04,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:05,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:06,618 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:37:06,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:07,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:08,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:09,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:37:09,453 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:37:09,454 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:37:09,455 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:37:09,567 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:37:09,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:10,568 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:37:10,702 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:11,776 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:12,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:37:12,762 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:13,762 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:14,762 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:15,762 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:16,394 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:37:16,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:17,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:37:17,495 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:37:17,559 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:37:17,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:18,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:19,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:20,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:21,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:22,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:37:22,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:22,872 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:37:22,874 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:37:22,875 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:37:22,876 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:37:23,578 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:37:23,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:24,579 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:37:24,763 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:25,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:26,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:27,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:27,916 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:37:28,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:29,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:30,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:31,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:32,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:37:32,496 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:37:32,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:33,638 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:37:33,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:34,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:35,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:36,328 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:37:36,329 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:37:36,330 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:37:36,331 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:37:36,587 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:37:36,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:37,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:38,589 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:37:38,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:39,369 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:37:39,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:40,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:41,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:42,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:43,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:44,370 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:37:44,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:45,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:46,396 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:37:46,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:47,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:37:47,496 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:37:47,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:48,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:49,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:37:49,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:49,854 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:37:49,856 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:37:49,857 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:37:49,858 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:37:50,597 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:37:50,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:51,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:52,599 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:37:52,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:53,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:54,896 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:37:54,906 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:55,905 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:56,905 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:57,905 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:58,905 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:37:59,897 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:37:59,905 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:00,906 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:01,906 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:02,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:38:02,496 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:38:02,906 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:03,143 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:38:03,145 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:38:03,145 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:38:03,147 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:38:03,607 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:38:03,906 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:04,608 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:38:04,906 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:05,185 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:38:05,906 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:06,906 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:07,906 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:08,907 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:09,907 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:10,186 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:38:10,907 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:11,907 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:12,907 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:13,907 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:14,907 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:15,187 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:38:15,907 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:16,400 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:38:16,595 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:38:16,597 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:38:16,597 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:38:16,599 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:38:16,617 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:38:16,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:17,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:38:17,496 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:38:17,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:18,618 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:38:18,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:19,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:20,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:38:20,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:21,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:22,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:23,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:24,908 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:25,569 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:38:25,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:26,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:27,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:28,909 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:29,903 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:38:29,905 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:38:29,905 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:38:29,907 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:38:30,144 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:30,628 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:38:30,945 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:38:31,039 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:32,039 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:32,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:38:32,496 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:38:32,629 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:38:33,039 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:34,039 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:35,039 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:36,039 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:36,567 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:38:37,039 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:38,040 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:39,040 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:40,040 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:41,040 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:41,568 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:38:42,040 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:43,040 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:43,301 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:38:43,303 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:38:43,303 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:38:43,305 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:38:43,637 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:38:44,040 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:44,638 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:38:45,040 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:46,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:46,404 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:38:47,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:47,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:38:47,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:38:47,497 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:38:48,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:49,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:50,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:51,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:52,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:52,647 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:38:53,041 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:54,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:55,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:56,042 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:56,873 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:38:56,875 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:38:56,875 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:38:56,877 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:38:57,116 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:57,651 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:38:57,915 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:38:58,113 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:38:58,652 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:38:59,203 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:00,188 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:01,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:02,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:02,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:39:02,496 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:39:03,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:03,654 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:39:04,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:05,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:06,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:07,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:08,189 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:08,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:39:09,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:10,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:10,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:39:10,240 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:39:10,241 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:39:10,242 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:39:10,661 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:39:11,191 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:12,191 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:12,663 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:39:13,191 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:14,191 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:14,281 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:39:15,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:16,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:16,407 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:39:17,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:17,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:39:17,497 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:39:18,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:19,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:19,566 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:39:20,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:21,256 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:22,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:23,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:23,597 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:39:23,599 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:39:23,599 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:39:23,601 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:39:23,671 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:39:24,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:24,639 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:39:24,672 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:39:25,257 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:26,258 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:27,284 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:28,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:29,266 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:29,639 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:39:30,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:31,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:32,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:32,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:39:32,497 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:39:33,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:34,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:35,270 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:35,584 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:39:36,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:37,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:37,355 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:39:37,356 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:39:37,357 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:39:37,358 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:39:37,681 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:39:38,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:38,682 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:39:39,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:40,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:41,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:41,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:39:42,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:43,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:44,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:45,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:46,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:46,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:39:46,411 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:39:47,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:47,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:39:47,497 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:39:48,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:49,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:50,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:50,773 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:39:50,775 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:39:50,776 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:39:50,777 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:39:51,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:51,692 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:39:51,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:39:52,272 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:52,693 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:39:53,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:54,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:55,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:56,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:56,815 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:39:57,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:58,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:39:59,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:00,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:01,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:01,816 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:40:02,273 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:02,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:40:02,497 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:40:03,274 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:04,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:40:04,206 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:40:04,206 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:40:04,208 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:40:04,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:04,702 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:40:05,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:06,518 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:06,704 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:40:07,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:40:07,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:08,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:09,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:10,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:11,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:12,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:40:12,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:13,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:14,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:15,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:16,416 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:40:16,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:17,418 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:40:17,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:40:17,497 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:40:17,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:17,620 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:40:17,656 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:40:17,657 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:40:17,658 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:40:17,712 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:40:18,510 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:18,713 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:40:19,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:20,510 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:21,510 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:22,510 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:22,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:40:23,510 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:24,510 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:25,510 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:26,510 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:27,510 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:27,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:40:28,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:29,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:30,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:31,065 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:40:31,067 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:40:31,068 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:40:31,069 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:40:31,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:31,723 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:40:32,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:40:32,497 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:40:32,537 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:32,723 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:40:33,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:33,610 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:40:34,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:35,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:36,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:37,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:38,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:38,610 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:40:39,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:40,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:41,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:42,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:43,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:43,611 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:40:44,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:45,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:46,418 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:40:46,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:47,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:40:47,516 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:40:47,556 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:48,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:48,616 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:40:49,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:50,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:51,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:52,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:53,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:53,617 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:40:54,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:55,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:56,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:40:58,738 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:40:59,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:40:59,158 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:04,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:41:05,488 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:41:05,917 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:07,022 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:41:07,023 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:41:07,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:09,148 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:41:14,080 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:41:14,149 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:41:14,562 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:15,841 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:16,422 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:41:16,510 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:19,424 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:41:22,549 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:41:23,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:41:23,637 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:41:24,704 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:41:24,836 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:29,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:41:32,422 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:41:33,021 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:34,001 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:34,705 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:41:39,706 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:41:41,200 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:41:41,846 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:42,150 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:41:42,151 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:41:43,207 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:45,267 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:41:46,426 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:41:50,428 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:41:54,712 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:41:54,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:54,859 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:55,754 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:41:55,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:56,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:41:56,337 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:41:56,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:57,047 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:41:57,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:58,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:41:59,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:00,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:01,466 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:42:01,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:02,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:03,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:04,860 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:05,861 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:06,466 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:42:06,861 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:07,861 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:08,861 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:09,861 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:10,861 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:11,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:42:11,337 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:42:11,506 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:42:11,861 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:12,861 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:13,868 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:14,867 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:15,867 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:16,429 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:42:16,867 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:17,431 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:42:17,868 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:18,868 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:19,868 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:20,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:21,888 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:22,432 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:42:22,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:23,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:24,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:25,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:26,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:42:26,337 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:42:26,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:27,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:42:27,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:28,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:29,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:30,889 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:31,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:32,497 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:42:32,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:33,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:34,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:35,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:36,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:37,498 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:42:37,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:38,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:39,890 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:40,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:41,336 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:42:41,337 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:42:41,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:42,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:43,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:42:43,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:44,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:45,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:46,433 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:42:46,891 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:47,892 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:48,892 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:49,435 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:42:49,892 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:50,892 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:42:54,436 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:42:59,437 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:43:04,437 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:43:09,438 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:43:13,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:43:13,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:43:13,282 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:13,283 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:43:14,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:15,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:15,389 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:43:16,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:16,437 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:43:17,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:18,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:19,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:20,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:20,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:43:21,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:22,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:23,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:24,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:25,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:25,440 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:43:26,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:27,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:28,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:43:28,238 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:43:28,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:29,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:30,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:31,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:31,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:43:32,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:33,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:34,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:35,239 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:36,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:36,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:43:37,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:38,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:39,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:40,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:41,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:41,376 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:43:42,240 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:43,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:43:43,238 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:43:43,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:44,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:45,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:46,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:46,402 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:43:46,440 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:43:47,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:48,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:49,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:50,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:51,241 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:51,442 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:43:52,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:53,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:54,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:55,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:56,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:56,443 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:43:57,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:58,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:43:58,238 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:43:58,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:43:59,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:00,242 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:01,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:02,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:02,310 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:44:03,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:04,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:05,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:06,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:07,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:07,311 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:44:08,243 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:09,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:10,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:11,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:12,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:12,312 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:44:13,237 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:44:13,238 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:44:13,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:14,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:15,244 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:16,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:16,443 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:44:17,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:17,446 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:44:18,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:19,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:20,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:21,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:22,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:22,447 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:44:23,245 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:24,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:25,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:26,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:27,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:27,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:44:28,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:44:28,238 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:44:28,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:29,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:30,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:31,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:32,246 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:33,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:33,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:44:34,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:35,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:36,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:37,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:38,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:38,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:44:39,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:40,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:41,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:42,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:43,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:44:43,238 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:44:43,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:44,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:44,327 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:44:45,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:46,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:46,447 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:44:47,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:48,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:49,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:49,449 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:44:50,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:51,248 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:52,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:53,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:54,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:54,450 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:44:55,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:56,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:57,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:58,238 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:44:58,238 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:44:58,278 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:44:59,249 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:00,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:00,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:45:01,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:02,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:03,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:04,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:05,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:05,406 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:45:06,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:07,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:08,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:09,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:10,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:10,407 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:45:11,251 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:15,408 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:45:16,450 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:45:20,453 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:45:25,458 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:45:30,454 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:45:35,455 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:45:40,458 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:45:45,459 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:45:46,454 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:45:47,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:45:47,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:47,397 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:45:47,398 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:45:48,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:49,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:50,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:50,735 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:45:50,898 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:45:51,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:52,361 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:52,900 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:45:53,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:54,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:55,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:55,922 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:45:56,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:57,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:58,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:45:59,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:00,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:00,923 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:46:01,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:02,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:46:02,360 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:46:02,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:03,332 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:46:03,334 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:46:03,335 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:46:03,336 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:46:03,582 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:03,907 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:46:04,581 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:04,908 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:46:05,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:06,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:46:06,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:07,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:08,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:09,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:10,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:11,374 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:46:11,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:12,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:13,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:14,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:15,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:16,375 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:46:16,459 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:46:16,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:16,772 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:46:16,774 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:46:16,775 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:46:16,777 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:46:16,915 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:46:17,360 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:46:17,361 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:46:17,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:18,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:18,917 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:46:19,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:20,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:21,504 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:46:21,658 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:22,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:23,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:24,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:25,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:26,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:46:26,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:27,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:28,659 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:29,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:30,163 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:46:30,165 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:46:30,165 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:46:30,167 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:46:30,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:30,925 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:46:30,926 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:46:31,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:32,205 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:46:32,477 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:46:32,478 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:46:32,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:33,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:34,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:35,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:36,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:37,560 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:46:37,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:38,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:39,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:40,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:41,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:42,561 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:46:42,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:43,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:46:43,631 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:46:43,631 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:46:43,633 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:46:43,869 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:43,934 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:46:44,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:44,935 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:46:45,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:46,461 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:46:46,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:47,463 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:46:47,463 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:46:47,575 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:46:47,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:48,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:49,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:50,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:51,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:52,576 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:46:52,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:53,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:54,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:55,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:56,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:57,322 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:46:57,323 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:46:57,324 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:46:57,327 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:46:57,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:57,944 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:46:58,363 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:46:58,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:46:58,945 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:46:59,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:00,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:01,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:02,463 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:47:02,463 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:47:02,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:03,575 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:47:03,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:04,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:05,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:06,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:07,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:08,576 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:47:08,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:09,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:10,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:10,887 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:47:10,888 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:47:10,889 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:47:10,890 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:47:10,954 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:47:11,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:12,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:12,956 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:47:13,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:13,930 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:47:14,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:15,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:16,465 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:47:16,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:17,463 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:47:17,464 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:47:17,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:18,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:19,624 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:47:19,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:20,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:21,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:22,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:23,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:24,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:47:24,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:25,772 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:26,777 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:27,776 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:28,776 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:29,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:47:29,776 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:30,776 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:31,777 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:32,463 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:47:32,463 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:47:32,777 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:33,777 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:34,777 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:35,571 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:47:35,777 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:36,777 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:38,975 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:47:39,168 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:41,082 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:47:45,359 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:47:46,082 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:47:46,410 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:46,468 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:47:51,471 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:47:53,875 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:47:54,517 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:47:54,518 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:47:54,814 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:56,171 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:47:56,614 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:48:01,615 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:48:03,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:48:03,848 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:04,023 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:48:04,023 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:48:04,822 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:07,190 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:48:11,924 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:12,191 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:48:13,802 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:16,472 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:48:17,474 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:48:20,448 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:48:20,931 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:48:20,932 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:48:21,062 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:21,989 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:23,059 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:48:23,275 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:28,060 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:48:33,061 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:48:35,230 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:48:35,271 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:35,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:35,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:48:35,622 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:48:36,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:37,237 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:48:37,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:38,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:38,436 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:48:39,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:40,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:41,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:42,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:43,353 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:43,437 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:48:44,354 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:45,354 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:46,354 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:46,475 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:48:47,354 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:48,354 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:48,478 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:48:49,354 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:50,354 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:50,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:48:50,622 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:48:51,355 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:52,355 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:53,355 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:53,722 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:48:54,355 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:55,355 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:56,355 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:57,355 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:58,355 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:48:58,723 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:48:59,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:00,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:01,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:02,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:03,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:03,724 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:49:04,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:05,356 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:05,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:49:05,622 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:49:06,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:07,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:08,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:09,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:09,696 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:49:10,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:11,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:12,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:13,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:14,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:14,697 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:49:15,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:16,357 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:16,480 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:49:17,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:18,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:19,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:20,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:20,482 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:49:20,622 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:49:20,622 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:49:21,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:22,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:23,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:24,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:25,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:25,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:49:26,358 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:27,359 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:28,359 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:29,359 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:30,359 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:30,716 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:49:31,359 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:32,359 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:35,717 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:49:40,717 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:49:45,718 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:49:46,483 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:49:51,486 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:49:53,621 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:49:53,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:53,662 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:49:53,663 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:49:54,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:55,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:56,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:56,739 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:49:57,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:58,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:49:59,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:00,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:01,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:01,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:50:02,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:03,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:04,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:05,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:06,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:06,740 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:50:07,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:08,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:50:08,626 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:50:08,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:09,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:10,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:11,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:11,762 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:50:12,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:13,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:14,627 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:15,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:16,486 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:50:16,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:17,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:50:17,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:18,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:19,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:20,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:21,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:22,489 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:50:22,628 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:23,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:50:23,626 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:50:23,666 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:24,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:25,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:26,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:27,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:27,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:50:28,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:29,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:30,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:31,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:32,629 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:32,768 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:50:33,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:34,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:35,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:36,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:37,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:37,769 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:50:38,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:50:38,626 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:50:38,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:39,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:40,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:41,630 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:42,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:42,790 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:50:43,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:44,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:45,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:46,490 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:50:46,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:47,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:48,492 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:50:48,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:49,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:50,631 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:51,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:52,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:53,493 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:50:53,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:50:53,626 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:50:53,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:54,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:55,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:56,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:57,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:58,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:50:58,751 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:50:59,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:00,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:01,632 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:02,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:03,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:03,752 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:51:04,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:05,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:06,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:07,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:08,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:51:08,626 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:51:08,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:09,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:09,697 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:51:10,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:11,633 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:12,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:13,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:14,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:14,697 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:51:15,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:16,494 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:51:16,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:17,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:18,634 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:19,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:20,496 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:51:20,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:21,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:22,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:23,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:51:23,626 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:51:23,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:24,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:25,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:25,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:51:26,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:27,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:28,635 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:29,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:30,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:30,720 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:51:31,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:32,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:33,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:34,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:35,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:35,721 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:51:36,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:37,636 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:38,626 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:51:38,627 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:51:38,667 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:39,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:40,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:41,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:41,711 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:51:42,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:43,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:44,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:45,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:46,497 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:51:46,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:47,499 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:51:47,637 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:51:52,500 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:51:57,501 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:52:02,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:52:07,502 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:52:12,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:52:16,501 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:52:17,503 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:52:22,506 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:52:27,505 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:52:27,770 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:52:27,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:27,810 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:52:27,811 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:52:28,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:29,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:30,606 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: partial_history 2025-01-05 00:52:30,606 DEBUG SenderThread:2321186 [sender.py:send():379] send: telemetry 2025-01-05 00:52:30,607 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:52:30,609 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:52:30,610 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:52:30,611 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:52:30,611 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:52:30,612 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:52:30,613 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:52:30,614 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:52:30,615 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:52:30,616 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:52:30,616 DEBUG SenderThread:2321186 [sender.py:send():379] send: history 2025-01-05 00:52:30,616 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: summary_record 2025-01-05 00:52:30,617 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:52:30,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:31,016 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:52:31,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:32,017 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:52:32,618 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:52:32,764 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:33,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:34,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:35,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:36,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:37,619 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:52:37,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:38,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:39,765 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:40,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:41,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:42,625 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:52:42,771 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:42,788 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:52:42,789 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:52:43,025 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/config.yaml 2025-01-05 00:52:43,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:44,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:45,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:46,506 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:52:46,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:47,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:48,508 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:52:48,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:49,766 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:50,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:51,767 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:52,783 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:53,509 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:52:53,781 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:54,782 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:55,782 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:52:58,036 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:52:59,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:52:59,247 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:53:04,204 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:53:05,538 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:53:05,956 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:53:05,956 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:53:06,396 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:53:07,318 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:53:10,047 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:53:13,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:53:14,775 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:53:15,048 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:53:15,409 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:53:15,410 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:53:16,425 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:53:16,508 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:53:20,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:53:23,250 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:53:24,182 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:53:25,511 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:53:25,718 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:53:30,512 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:53:32,695 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:53:33,613 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:53:33,919 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:53:33,920 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:53:34,554 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:53:36,053 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:53:41,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:53:41,698 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:53:42,132 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:53:43,481 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:53:44,804 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:53:46,054 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:53:46,512 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:53:51,452 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: keepalive 2025-01-05 00:53:51,514 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:53:56,515 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:53:56,574 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:53:56,574 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: stop_status 2025-01-05 00:53:56,576 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: stop_status 2025-01-05 00:53:56,581 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:53:57,577 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:53:58,152 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:53:58,577 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:53:58,729 DEBUG SenderThread:2321186 [sender.py:send():379] send: exit 2025-01-05 00:53:58,730 INFO SenderThread:2321186 [sender.py:send_exit():586] handling exit code: 0 2025-01-05 00:53:58,730 INFO SenderThread:2321186 [sender.py:send_exit():588] handling runtime: 15644 2025-01-05 00:53:58,732 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:53:58,732 INFO SenderThread:2321186 [sender.py:send_exit():594] send defer 2025-01-05 00:53:58,733 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: defer 2025-01-05 00:53:58,733 INFO HandlerThread:2321186 [handler.py:handle_request_defer():184] handle defer: 0 2025-01-05 00:53:58,733 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: defer 2025-01-05 00:53:58,733 INFO SenderThread:2321186 [sender.py:send_request_defer():610] handle sender defer: 0 2025-01-05 00:53:58,733 INFO SenderThread:2321186 [sender.py:transition_state():614] send defer: 1 2025-01-05 00:53:58,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: defer 2025-01-05 00:53:58,734 INFO HandlerThread:2321186 [handler.py:handle_request_defer():184] handle defer: 1 2025-01-05 00:53:58,734 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: defer 2025-01-05 00:53:58,734 INFO SenderThread:2321186 [sender.py:send_request_defer():610] handle sender defer: 1 2025-01-05 00:53:58,734 INFO SenderThread:2321186 [sender.py:transition_state():614] send defer: 2 2025-01-05 00:53:58,734 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: defer 2025-01-05 00:53:58,734 INFO HandlerThread:2321186 [handler.py:handle_request_defer():184] handle defer: 2 2025-01-05 00:53:58,734 INFO HandlerThread:2321186 [system_monitor.py:finish():203] Stopping system monitor 2025-01-05 00:53:58,735 DEBUG SystemMonitor:2321186 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2025-01-05 00:53:58,735 DEBUG SystemMonitor:2321186 [system_monitor.py:_start():183] Publishing last batch of metrics 2025-01-05 00:53:58,739 INFO HandlerThread:2321186 [interfaces.py:finish():200] Joined cpu monitor 2025-01-05 00:53:58,739 INFO HandlerThread:2321186 [interfaces.py:finish():200] Joined disk monitor 2025-01-05 00:53:59,153 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:53:59,652 INFO HandlerThread:2321186 [interfaces.py:finish():200] Joined gpu monitor 2025-01-05 00:53:59,652 INFO HandlerThread:2321186 [interfaces.py:finish():200] Joined memory monitor 2025-01-05 00:53:59,653 INFO HandlerThread:2321186 [interfaces.py:finish():200] Joined network monitor 2025-01-05 00:53:59,654 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: defer 2025-01-05 00:53:59,654 INFO SenderThread:2321186 [sender.py:send_request_defer():610] handle sender defer: 2 2025-01-05 00:53:59,654 INFO SenderThread:2321186 [sender.py:transition_state():614] send defer: 3 2025-01-05 00:53:59,654 DEBUG SenderThread:2321186 [sender.py:send():379] send: stats 2025-01-05 00:53:59,655 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: defer 2025-01-05 00:53:59,655 INFO HandlerThread:2321186 [handler.py:handle_request_defer():184] handle defer: 3 2025-01-05 00:53:59,656 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: defer 2025-01-05 00:53:59,656 INFO SenderThread:2321186 [sender.py:send_request_defer():610] handle sender defer: 3 2025-01-05 00:53:59,656 INFO SenderThread:2321186 [sender.py:transition_state():614] send defer: 4 2025-01-05 00:53:59,656 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: defer 2025-01-05 00:53:59,656 INFO HandlerThread:2321186 [handler.py:handle_request_defer():184] handle defer: 4 2025-01-05 00:53:59,657 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: defer 2025-01-05 00:53:59,657 INFO SenderThread:2321186 [sender.py:send_request_defer():610] handle sender defer: 4 2025-01-05 00:53:59,657 INFO SenderThread:2321186 [sender.py:transition_state():614] send defer: 5 2025-01-05 00:53:59,657 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: defer 2025-01-05 00:53:59,657 INFO HandlerThread:2321186 [handler.py:handle_request_defer():184] handle defer: 5 2025-01-05 00:53:59,658 DEBUG SenderThread:2321186 [sender.py:send():379] send: summary 2025-01-05 00:53:59,659 INFO SenderThread:2321186 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-05 00:53:59,660 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: defer 2025-01-05 00:53:59,660 INFO SenderThread:2321186 [sender.py:send_request_defer():610] handle sender defer: 5 2025-01-05 00:53:59,660 INFO SenderThread:2321186 [sender.py:transition_state():614] send defer: 6 2025-01-05 00:53:59,660 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: defer 2025-01-05 00:53:59,660 INFO HandlerThread:2321186 [handler.py:handle_request_defer():184] handle defer: 6 2025-01-05 00:53:59,660 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: defer 2025-01-05 00:53:59,661 INFO SenderThread:2321186 [sender.py:send_request_defer():610] handle sender defer: 6 2025-01-05 00:53:59,661 INFO SenderThread:2321186 [sender.py:transition_state():614] send defer: 7 2025-01-05 00:53:59,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: status_report 2025-01-05 00:53:59,661 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: defer 2025-01-05 00:53:59,661 INFO HandlerThread:2321186 [handler.py:handle_request_defer():184] handle defer: 7 2025-01-05 00:53:59,661 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: defer 2025-01-05 00:53:59,662 INFO SenderThread:2321186 [sender.py:send_request_defer():610] handle sender defer: 7 2025-01-05 00:53:59,730 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-05 00:54:00,154 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:54:00,155 INFO Thread-12 :2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:54:03,216 INFO SenderThread:2321186 [sender.py:transition_state():614] send defer: 8 2025-01-05 00:54:03,217 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: poll_exit 2025-01-05 00:54:03,217 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: defer 2025-01-05 00:54:03,217 INFO HandlerThread:2321186 [handler.py:handle_request_defer():184] handle defer: 8 2025-01-05 00:54:03,218 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: defer 2025-01-05 00:54:03,218 INFO SenderThread:2321186 [sender.py:send_request_defer():610] handle sender defer: 8 2025-01-05 00:54:03,218 INFO SenderThread:2321186 [job_builder.py:build():440] Attempting to build job artifact 2025-01-05 00:54:03,222 INFO SenderThread:2321186 [job_builder.py:_get_source_type():569] is repo sourced job 2025-01-05 00:54:03,470 INFO SenderThread:2321186 [job_builder.py:build():545] adding wandb-job metadata file 2025-01-05 00:54:03,494 INFO SenderThread:2321186 [sender.py:transition_state():614] send defer: 9 2025-01-05 00:54:03,495 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: defer 2025-01-05 00:54:03,495 INFO HandlerThread:2321186 [handler.py:handle_request_defer():184] handle defer: 9 2025-01-05 00:54:03,495 DEBUG SenderThread:2321186 [sender.py:send():379] send: artifact 2025-01-05 00:54:03,731 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-05 00:54:03,834 INFO SenderThread:2321186 [sender.py:send_artifact():1530] sent artifact job-https___github.com_chtmp223_BookGen-dev.git_prolong-final_finetune.py - {'id': 'QXJ0aWZhY3Q6MTQxODc0MDI4NQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjQ5NzE2NzEyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxODc0MDI4NQ==', 'versionIndex': 25}}} 2025-01-05 00:54:03,834 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: defer 2025-01-05 00:54:03,834 INFO SenderThread:2321186 [sender.py:send_request_defer():610] handle sender defer: 9 2025-01-05 00:54:03,835 INFO SenderThread:2321186 [dir_watcher.py:finish():358] shutting down directory watcher 2025-01-05 00:54:04,157 INFO SenderThread:2321186 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:54:04,158 INFO SenderThread:2321186 [dir_watcher.py:finish():388] scan: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files 2025-01-05 00:54:04,158 INFO SenderThread:2321186 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/conda-environment.yaml conda-environment.yaml 2025-01-05 00:54:04,158 INFO SenderThread:2321186 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json wandb-summary.json 2025-01-05 00:54:04,159 INFO SenderThread:2321186 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-metadata.json wandb-metadata.json 2025-01-05 00:54:04,159 INFO SenderThread:2321186 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log output.log 2025-01-05 00:54:04,165 INFO SenderThread:2321186 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/config.yaml config.yaml 2025-01-05 00:54:04,166 INFO SenderThread:2321186 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/requirements.txt requirements.txt 2025-01-05 00:54:04,166 INFO SenderThread:2321186 [sender.py:transition_state():614] send defer: 10 2025-01-05 00:54:04,166 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: poll_exit 2025-01-05 00:54:04,166 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: defer 2025-01-05 00:54:04,167 INFO HandlerThread:2321186 [handler.py:handle_request_defer():184] handle defer: 10 2025-01-05 00:54:04,168 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: defer 2025-01-05 00:54:04,168 INFO SenderThread:2321186 [sender.py:send_request_defer():610] handle sender defer: 10 2025-01-05 00:54:04,168 INFO SenderThread:2321186 [file_pusher.py:finish():169] shutting down file pusher 2025-01-05 00:54:04,483 INFO wandb-upload_3:2321186 [upload_job.py:push():130] Uploaded file /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/config.yaml 2025-01-05 00:54:04,485 INFO wandb-upload_1:2321186 [upload_job.py:push():130] Uploaded file /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/wandb-summary.json 2025-01-05 00:54:04,485 INFO wandb-upload_0:2321186 [upload_job.py:push():130] Uploaded file /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/conda-environment.yaml 2025-01-05 00:54:04,485 INFO wandb-upload_4:2321186 [upload_job.py:push():130] Uploaded file /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/requirements.txt 2025-01-05 00:54:04,513 INFO wandb-upload_2:2321186 [upload_job.py:push():130] Uploaded file /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/files/output.log 2025-01-05 00:54:04,714 INFO Thread-11 (_thread_body):2321186 [sender.py:transition_state():614] send defer: 11 2025-01-05 00:54:04,715 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: defer 2025-01-05 00:54:04,715 INFO HandlerThread:2321186 [handler.py:handle_request_defer():184] handle defer: 11 2025-01-05 00:54:04,715 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: defer 2025-01-05 00:54:04,716 INFO SenderThread:2321186 [sender.py:send_request_defer():610] handle sender defer: 11 2025-01-05 00:54:04,716 INFO SenderThread:2321186 [file_pusher.py:join():175] waiting for file pusher 2025-01-05 00:54:04,716 INFO SenderThread:2321186 [sender.py:transition_state():614] send defer: 12 2025-01-05 00:54:04,717 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: defer 2025-01-05 00:54:04,717 INFO HandlerThread:2321186 [handler.py:handle_request_defer():184] handle defer: 12 2025-01-05 00:54:04,717 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: defer 2025-01-05 00:54:04,717 INFO SenderThread:2321186 [sender.py:send_request_defer():610] handle sender defer: 12 2025-01-05 00:54:04,717 INFO SenderThread:2321186 [file_stream.py:finish():601] file stream finish called 2025-01-05 00:54:04,732 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-05 00:54:04,945 INFO SenderThread:2321186 [file_stream.py:finish():605] file stream finish is done 2025-01-05 00:54:04,945 INFO SenderThread:2321186 [sender.py:transition_state():614] send defer: 13 2025-01-05 00:54:04,945 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: poll_exit 2025-01-05 00:54:04,946 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: defer 2025-01-05 00:54:04,946 INFO HandlerThread:2321186 [handler.py:handle_request_defer():184] handle defer: 13 2025-01-05 00:54:04,946 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: defer 2025-01-05 00:54:04,946 INFO SenderThread:2321186 [sender.py:send_request_defer():610] handle sender defer: 13 2025-01-05 00:54:04,946 INFO SenderThread:2321186 [sender.py:transition_state():614] send defer: 14 2025-01-05 00:54:04,947 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: defer 2025-01-05 00:54:04,947 INFO HandlerThread:2321186 [handler.py:handle_request_defer():184] handle defer: 14 2025-01-05 00:54:04,947 DEBUG SenderThread:2321186 [sender.py:send():379] send: final 2025-01-05 00:54:04,947 DEBUG SenderThread:2321186 [sender.py:send():379] send: footer 2025-01-05 00:54:04,947 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: defer 2025-01-05 00:54:04,948 INFO SenderThread:2321186 [sender.py:send_request_defer():610] handle sender defer: 14 2025-01-05 00:54:04,948 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-05 00:54:04,949 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: poll_exit 2025-01-05 00:54:04,949 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: server_info 2025-01-05 00:54:04,950 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: server_info 2025-01-05 00:54:04,956 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-05 00:54:04,956 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: get_summary 2025-01-05 00:54:04,957 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: sampled_history 2025-01-05 00:54:04,961 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-05 00:54:05,012 DEBUG SenderThread:2321186 [sender.py:send_request():406] send_request: poll_exit 2025-01-05 00:54:05,012 INFO MainThread:2321186 [wandb_run.py:_footer_history_summary_info():3991] rendering history 2025-01-05 00:54:05,014 INFO MainThread:2321186 [wandb_run.py:_footer_history_summary_info():4023] rendering summary 2025-01-05 00:54:05,014 INFO MainThread:2321186 [wandb_run.py:_footer_sync_info():3950] logging synced files 2025-01-05 00:54:05,015 DEBUG HandlerThread:2321186 [handler.py:handle_request():158] handle_request: shutdown 2025-01-05 00:54:05,015 INFO HandlerThread:2321186 [handler.py:finish():882] shutting down handler 2025-01-05 00:54:05,956 INFO WriterThread:2321186 [datastore.py:close():296] close: /scratch3/workspace/ctpham_umass_edu-models/wp_hparam_prolong-512K-base_bsz-16_lr-1e-5_epochs-1_/wandb/run-20250104_203314-95wq5z4x/run-95wq5z4x.wandb 2025-01-05 00:54:06,012 INFO SenderThread:2321186 [sender.py:finish():1608] shutting down sender 2025-01-05 00:54:06,013 INFO SenderThread:2321186 [file_pusher.py:finish():169] shutting down file pusher 2025-01-05 00:54:06,013 INFO SenderThread:2321186 [file_pusher.py:join():175] waiting for file pusher