2024-05-25 19:26:38,715 INFO    MainThread:27183 [wandb_setup.py:_flush():76] Current SDK version is 0.17.0
2024-05-25 19:26:38,715 INFO    MainThread:27183 [wandb_setup.py:_flush():76] Configure stats pid to 27183
2024-05-25 19:26:38,715 INFO    MainThread:27183 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings
2024-05-25 19:26:38,715 INFO    MainThread:27183 [wandb_setup.py:_flush():76] Loading settings from /root/mistral-finetune/wandb/settings
2024-05-25 19:26:38,716 INFO    MainThread:27183 [wandb_setup.py:_flush():76] Loading settings from environment variables: {}
2024-05-25 19:26:38,716 WARNING MainThread:27183 [wandb_setup.py:_flush():76] Could not find program at -m train
2024-05-25 19:26:38,716 INFO    MainThread:27183 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': None, 'program': '-m train'}
2024-05-25 19:26:38,716 INFO    MainThread:27183 [wandb_setup.py:_flush():76] Applying login settings: {'api_key': '***REDACTED***'}
2024-05-25 19:26:38,716 INFO    MainThread:27183 [wandb_setup.py:_flush():76] Applying login settings: {'api_key': '***REDACTED***'}
2024-05-25 19:26:38,716 INFO    MainThread:27183 [wandb_setup.py:_flush():76] Applying login settings: {}
2024-05-25 19:26:38,716 INFO    MainThread:27183 [wandb_init.py:_log_setup():520] Logging user logs to /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/logs/debug.log
2024-05-25 19:26:38,716 INFO    MainThread:27183 [wandb_init.py:_log_setup():521] Logging internal logs to /root/mistral-finetune/runs/wandb/run-20240525_192638-zf4t8z6a/logs/debug-internal.log
2024-05-25 19:26:38,716 INFO    MainThread:27183 [wandb_init.py:init():560] calling init triggers
2024-05-25 19:26:38,716 INFO    MainThread:27183 [wandb_init.py:init():567] wandb.init called with sweep_config: {}
config: {'data': {'data': '', 'shuffle': False, 'instruct_data': '/root/data/mol_instructions_train.jsonl', 'eval_instruct_data': '', 'instruct': {'shuffle': True, 'dynamic_chunk_fn_call': True}}, 'model_id_or_path': '/root/mistral_models/7B-v0.3', 'run_dir': '/root/mistral-finetune/runs', 'optim': {'lr': 6e-05, 'weight_decay': 0.1, 'pct_start': 0.05}, 'seed': 0, 'num_microbatches': 1, 'seq_len': 32768, 'batch_size': 1, 'max_norm': 1.0, 'max_steps': 300, 'log_freq': 1, 'ckpt_freq': 100, 'ckpt_only_lora': True, 'no_ckpt': False, 'num_ckpt_keep': 3, 'eval_freq': 100, 'no_eval': True, 'checkpoint': True, 'world_size': 1, 'wandb': {'project': 'CHEMISTral7b-ft', 'offline': False, 'key': 'aaf77f83a4e316f6a8b47fa975ab6b5e73c7c8df', 'run_name': 'run0'}, 'mlflow': {'tracking_uri': None, 'experiment_name': None}, 'lora': {'enable': True, 'rank': 64, 'dropout': 0.0, 'scaling': 2.0}}
2024-05-25 19:26:38,716 INFO    MainThread:27183 [wandb_init.py:init():610] starting backend
2024-05-25 19:26:38,716 INFO    MainThread:27183 [wandb_init.py:init():614] setting up manager
2024-05-25 19:26:38,716 INFO    MainThread:27183 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2024-05-25 19:26:38,717 INFO    MainThread:27183 [wandb_init.py:init():622] backend started and connected
2024-05-25 19:26:38,718 INFO    MainThread:27183 [wandb_init.py:init():711] updated telemetry
2024-05-25 19:26:38,720 INFO    MainThread:27183 [wandb_init.py:init():744] communicating run to backend with 90.0 second timeout
2024-05-25 19:26:39,028 INFO    MainThread:27183 [wandb_run.py:_on_init():2396] communicating current version
2024-05-25 19:26:39,057 INFO    MainThread:27183 [wandb_run.py:_on_init():2405] got version response 
2024-05-25 19:26:39,058 INFO    MainThread:27183 [wandb_init.py:init():795] starting run threads in backend
2024-05-25 19:26:40,070 INFO    MainThread:27183 [wandb_run.py:_console_start():2374] atexit reg
2024-05-25 19:26:40,070 INFO    MainThread:27183 [wandb_run.py:_redirect():2229] redirect: wrap_raw
2024-05-25 19:26:40,070 INFO    MainThread:27183 [wandb_run.py:_redirect():2294] Wrapping output streams.
2024-05-25 19:26:40,070 INFO    MainThread:27183 [wandb_run.py:_redirect():2319] Redirects installed.
2024-05-25 19:26:40,071 INFO    MainThread:27183 [wandb_init.py:init():838] run started, returning control to user process
2024-05-25 20:08:38,614 INFO    MainThread:27183 [wandb_run.py:_finish():2103] finishing run artsy/CHEMISTral7b-ft/zf4t8z6a
2024-05-25 20:08:38,615 INFO    MainThread:27183 [wandb_run.py:_atexit_cleanup():2343] got exitcode: 0
2024-05-25 20:08:38,615 INFO    MainThread:27183 [wandb_run.py:_restore():2326] restore
2024-05-25 20:08:38,615 INFO    MainThread:27183 [wandb_run.py:_restore():2332] restore done
2024-05-25 20:08:41,738 INFO    MainThread:27183 [wandb_run.py:_footer_history_summary_info():3994] rendering history
2024-05-25 20:08:41,738 INFO    MainThread:27183 [wandb_run.py:_footer_history_summary_info():4026] rendering summary
2024-05-25 20:08:41,741 INFO    MainThread:27183 [wandb_run.py:_footer_sync_info():3953] logging synced files