| 2024-05-26 01:46:34,619 INFO MainThread:66405 [wandb_setup.py:_flush():76] Current SDK version is 0.17.0 | |
| 2024-05-26 01:46:34,619 INFO MainThread:66405 [wandb_setup.py:_flush():76] Configure stats pid to 66405 | |
| 2024-05-26 01:46:34,619 INFO MainThread:66405 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings | |
| 2024-05-26 01:46:34,619 INFO MainThread:66405 [wandb_setup.py:_flush():76] Loading settings from /root/mistral-finetune/wandb/settings | |
| 2024-05-26 01:46:34,619 INFO MainThread:66405 [wandb_setup.py:_flush():76] Loading settings from environment variables: {} | |
| 2024-05-26 01:46:34,619 WARNING MainThread:66405 [wandb_setup.py:_flush():76] Could not find program at -m train | |
| 2024-05-26 01:46:34,619 INFO MainThread:66405 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': None, 'program': '-m train'} | |
| 2024-05-26 01:46:34,619 INFO MainThread:66405 [wandb_setup.py:_flush():76] Applying login settings: {'api_key': '***REDACTED***'} | |
| 2024-05-26 01:46:34,619 INFO MainThread:66405 [wandb_setup.py:_flush():76] Applying login settings: {'api_key': '***REDACTED***'} | |
| 2024-05-26 01:46:34,619 INFO MainThread:66405 [wandb_setup.py:_flush():76] Applying login settings: {} | |
| 2024-05-26 01:46:34,619 INFO MainThread:66405 [wandb_init.py:_log_setup():520] Logging user logs to /root/mistral-finetune/runseed99/wandb/run-20240526_014634-8idqp8iz/logs/debug.log | |
| 2024-05-26 01:46:34,619 INFO MainThread:66405 [wandb_init.py:_log_setup():521] Logging internal logs to /root/mistral-finetune/runseed99/wandb/run-20240526_014634-8idqp8iz/logs/debug-internal.log | |
| 2024-05-26 01:46:34,619 INFO MainThread:66405 [wandb_init.py:init():560] calling init triggers | |
| 2024-05-26 01:46:34,619 INFO MainThread:66405 [wandb_init.py:init():567] wandb.init called with sweep_config: {} | |
| config: {'data': {'data': '', 'shuffle': False, 'instruct_data': '/root/data/mol_instructions_train.jsonl', 'eval_instruct_data': '', 'instruct': {'shuffle': True, 'dynamic_chunk_fn_call': True}}, 'model_id_or_path': '/root/mistral_models/7B-v0.3', 'run_dir': '/root/mistral-finetune/runseed99', 'optim': {'lr': 5e-05, 'weight_decay': 0.05, 'pct_start': 0.05}, 'seed': 99, 'num_microbatches': 1, 'seq_len': 32768, 'batch_size': 2, 'max_norm': 1.0, 'max_steps': 500, 'log_freq': 1, 'ckpt_freq': 100, 'ckpt_only_lora': False, 'no_ckpt': False, 'num_ckpt_keep': 3, 'eval_freq': 100, 'no_eval': True, 'checkpoint': True, 'world_size': 1, 'wandb': {'project': 'CHEMISTral7b-ft', 'offline': False, 'key': 'aaf77f83a4e316f6a8b47fa975ab6b5e73c7c8df', 'run_name': 'runseed99'}, 'mlflow': {'tracking_uri': None, 'experiment_name': None}, 'lora': {'enable': True, 'rank': 64, 'dropout': 0.0, 'scaling': 2.0}} | |
| 2024-05-26 01:46:34,619 INFO MainThread:66405 [wandb_init.py:init():610] starting backend | |
| 2024-05-26 01:46:34,619 INFO MainThread:66405 [wandb_init.py:init():614] setting up manager | |
| 2024-05-26 01:46:34,620 INFO MainThread:66405 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=fork,spawn,forkserver, using: spawn | |
| 2024-05-26 01:46:34,621 INFO MainThread:66405 [wandb_init.py:init():622] backend started and connected | |
| 2024-05-26 01:46:34,622 INFO MainThread:66405 [wandb_init.py:init():711] updated telemetry | |
| 2024-05-26 01:46:34,624 INFO MainThread:66405 [wandb_init.py:init():744] communicating run to backend with 90.0 second timeout | |
| 2024-05-26 01:46:34,969 INFO MainThread:66405 [wandb_run.py:_on_init():2396] communicating current version | |
| 2024-05-26 01:46:34,984 INFO MainThread:66405 [wandb_run.py:_on_init():2405] got version response | |
| 2024-05-26 01:46:34,984 INFO MainThread:66405 [wandb_init.py:init():795] starting run threads in backend | |
| 2024-05-26 01:46:36,108 INFO MainThread:66405 [wandb_run.py:_console_start():2374] atexit reg | |
| 2024-05-26 01:46:36,109 INFO MainThread:66405 [wandb_run.py:_redirect():2229] redirect: wrap_raw | |
| 2024-05-26 01:46:36,109 INFO MainThread:66405 [wandb_run.py:_redirect():2294] Wrapping output streams. | |
| 2024-05-26 01:46:36,109 INFO MainThread:66405 [wandb_run.py:_redirect():2319] Redirects installed. | |
| 2024-05-26 01:46:36,109 INFO MainThread:66405 [wandb_init.py:init():838] run started, returning control to user process | |