| [2024-03-26 12:47:55,045] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect) | |
| [2024-03-26 12:47:56,486] [WARNING] [runner.py:202:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only. | |
| [2024-03-26 12:47:56,486] [INFO] [runner.py:568:main] cmd = /home/lirenhao/anaconda3/envs/llama_factory/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgM119 --master_addr=127.0.0.1 --master_port=2345 --enable_each_rank_log=None /home/lirenhao/projects/LLaMA-Factory/src/train_bash.py --deepspeed /home/lirenhao/projects/LLaMA-Factory/ds_configs/zero2_no_offload.json --stage sft --model_name_or_path /home/lirenhao/pretrained_models/Llama-2-7b-hf --do_train --dataset alpaca_random9k_evol-chatgpt --template llama2 --finetuning_type full --lora_target q_proj,v_proj --output_dir /home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909 --overwrite_cache --overwrite_output_dir --per_device_train_batch_size 32 --gradient_accumulation_steps 4 --lr_scheduler_type cosine --logging_steps 10 --save_steps 1000 --learning_rate 2e-5 --warmup_ratio 0.1 --num_train_epochs 3.0 --cutoff_len 4096 --plot_loss --bf16 | |
| [2024-03-26 12:47:58,430] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect) | |
| [2024-03-26 12:47:59,409] [INFO] [launch.py:145:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3]} | |
| [2024-03-26 12:47:59,409] [INFO] [launch.py:151:main] nnodes=1, num_local_procs=4, node_rank=0 | |
| [2024-03-26 12:47:59,409] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3]}) | |
| [2024-03-26 12:47:59,409] [INFO] [launch.py:163:main] dist_world_size=4 | |
| [2024-03-26 12:47:59,409] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3 | |
| [2024-03-26 12:48:02,978] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect) | |
| [2024-03-26 12:48:02,991] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect) | |
| [2024-03-26 12:48:03,011] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect) | |
| [2024-03-26 12:48:03,011] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect) | |
| [2024-03-26 12:48:05,044] [INFO] [comm.py:637:init_distributed] cdb=None | |
| [2024-03-26 12:48:05,063] [INFO] [comm.py:637:init_distributed] cdb=None | |
| [2024-03-26 12:48:05,063] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl | |
| [2024-03-26 12:48:05,092] [INFO] [comm.py:637:init_distributed] cdb=None | |
| [2024-03-26 12:48:05,095] [INFO] [comm.py:637:init_distributed] cdb=None | |
| 03/26/2024 12:48:05 - INFO - llmtuner.hparams.parser - Process rank: 1, device: cuda:1, n_gpu: 1 | |
| distributed training: True, compute dtype: torch.bfloat16 | |
| 03/26/2024 12:48:05 - INFO - llmtuner.hparams.parser - Training/evaluation parameters Seq2SeqTrainingArguments( | |
| _n_gpu=1, | |
| accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True}, | |
| adafactor=False, | |
| adam_beta1=0.9, | |
| adam_beta2=0.999, | |
| adam_epsilon=1e-08, | |
| auto_find_batch_size=False, | |
| bf16=True, | |
| bf16_full_eval=False, | |
| data_seed=None, | |
| dataloader_drop_last=False, | |
| dataloader_num_workers=0, | |
| dataloader_persistent_workers=False, | |
| dataloader_pin_memory=True, | |
| dataloader_prefetch_factor=None, | |
| ddp_backend=None, | |
| ddp_broadcast_buffers=None, | |
| ddp_bucket_cap_mb=None, | |
| ddp_find_unused_parameters=None, | |
| ddp_timeout=1800, | |
| debug=[], | |
| deepspeed=/home/lirenhao/projects/LLaMA-Factory/ds_configs/zero2_no_offload.json, | |
| disable_tqdm=False, | |
| dispatch_batches=None, | |
| do_eval=False, | |
| do_predict=False, | |
| do_train=True, | |
| eval_accumulation_steps=None, | |
| eval_delay=0, | |
| eval_steps=None, | |
| evaluation_strategy=no, | |
| fp16=False, | |
| fp16_backend=auto, | |
| fp16_full_eval=False, | |
| fp16_opt_level=O1, | |
| fsdp=[], | |
| fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, | |
| fsdp_min_num_params=0, | |
| fsdp_transformer_layer_cls_to_wrap=None, | |
| full_determinism=False, | |
| generation_config=None, | |
| generation_max_length=None, | |
| generation_num_beams=None, | |
| gradient_accumulation_steps=4, | |
| gradient_checkpointing=False, | |
| gradient_checkpointing_kwargs=None, | |
| greater_is_better=None, | |
| group_by_length=False, | |
| half_precision_backend=auto, | |
| hub_always_push=False, | |
| hub_model_id=None, | |
| hub_private_repo=False, | |
| hub_strategy=every_save, | |
| hub_token=<HUB_TOKEN>, | |
| ignore_data_skip=False, | |
| include_inputs_for_metrics=False, | |
| include_num_input_tokens_seen=False, | |
| include_tokens_per_second=False, | |
| jit_mode_eval=False, | |
| label_names=None, | |
| label_smoothing_factor=0.0, | |
| learning_rate=2e-05, | |
| length_column_name=length, | |
| load_best_model_at_end=False, | |
| local_rank=1, | |
| log_level=passive, | |
| log_level_replica=warning, | |
| log_on_each_node=True, | |
| logging_dir=/home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909/runs/Mar26_12-48-05_siat-a100-4-01, | |
| logging_first_step=False, | |
| logging_nan_inf_filter=True, | |
| logging_steps=10, | |
| logging_strategy=steps, | |
| lr_scheduler_kwargs={}, | |
| lr_scheduler_type=cosine, | |
| max_grad_norm=1.0, | |
| max_steps=-1, | |
| metric_for_best_model=None, | |
| mp_parameters=, | |
| neftune_noise_alpha=None, | |
| no_cuda=False, | |
| num_train_epochs=3.0, | |
| optim=adamw_torch, | |
| optim_args=None, | |
| output_dir=/home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909, | |
| overwrite_output_dir=True, | |
| past_index=-1, | |
| per_device_eval_batch_size=8, | |
| per_device_train_batch_size=32, | |
| predict_with_generate=False, | |
| prediction_loss_only=False, | |
| push_to_hub=False, | |
| push_to_hub_model_id=None, | |
| push_to_hub_organization=None, | |
| push_to_hub_token=<PUSH_TO_HUB_TOKEN>, | |
| ray_scope=last, | |
| remove_unused_columns=True, | |
| report_to=[], | |
| resume_from_checkpoint=None, | |
| run_name=/home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909, | |
| save_on_each_node=False, | |
| save_only_model=False, | |
| save_safetensors=True, | |
| save_steps=1000, | |
| save_strategy=steps, | |
| save_total_limit=None, | |
| seed=42, | |
| skip_memory_metrics=True, | |
| sortish_sampler=False, | |
| split_batches=None, | |
| tf32=None, | |
| torch_compile=False, | |
| torch_compile_backend=None, | |
| torch_compile_mode=None, | |
| torchdynamo=None, | |
| tpu_metrics_debug=False, | |
| tpu_num_cores=None, | |
| use_cpu=False, | |
| use_ipex=False, | |
| use_legacy_prediction_loop=False, | |
| use_mps_device=False, | |
| warmup_ratio=0.1, | |
| warmup_steps=0, | |
| weight_decay=0.0, | |
| ) | |
| 03/26/2024 12:48:05 - INFO - llmtuner.hparams.parser - Process rank: 2, device: cuda:2, n_gpu: 1 | |
| distributed training: True, compute dtype: torch.bfloat16 | |
| 03/26/2024 12:48:05 - INFO - llmtuner.hparams.parser - Training/evaluation parameters Seq2SeqTrainingArguments( | |
| _n_gpu=1, | |
| accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True}, | |
| adafactor=False, | |
| adam_beta1=0.9, | |
| adam_beta2=0.999, | |
| adam_epsilon=1e-08, | |
| auto_find_batch_size=False, | |
| bf16=True, | |
| bf16_full_eval=False, | |
| data_seed=None, | |
| dataloader_drop_last=False, | |
| dataloader_num_workers=0, | |
| dataloader_persistent_workers=False, | |
| dataloader_pin_memory=True, | |
| dataloader_prefetch_factor=None, | |
| ddp_backend=None, | |
| ddp_broadcast_buffers=None, | |
| ddp_bucket_cap_mb=None, | |
| ddp_find_unused_parameters=None, | |
| ddp_timeout=1800, | |
| debug=[], | |
| deepspeed=/home/lirenhao/projects/LLaMA-Factory/ds_configs/zero2_no_offload.json, | |
| disable_tqdm=False, | |
| dispatch_batches=None, | |
| do_eval=False, | |
| do_predict=False, | |
| do_train=True, | |
| eval_accumulation_steps=None, | |
| eval_delay=0, | |
| eval_steps=None, | |
| evaluation_strategy=no, | |
| fp16=False, | |
| fp16_backend=auto, | |
| fp16_full_eval=False, | |
| fp16_opt_level=O1, | |
| fsdp=[], | |
| fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, | |
| fsdp_min_num_params=0, | |
| fsdp_transformer_layer_cls_to_wrap=None, | |
| full_determinism=False, | |
| generation_config=None, | |
| generation_max_length=None, | |
| generation_num_beams=None, | |
| gradient_accumulation_steps=4, | |
| gradient_checkpointing=False, | |
| gradient_checkpointing_kwargs=None, | |
| greater_is_better=None, | |
| group_by_length=False, | |
| half_precision_backend=auto, | |
| hub_always_push=False, | |
| hub_model_id=None, | |
| hub_private_repo=False, | |
| hub_strategy=every_save, | |
| hub_token=<HUB_TOKEN>, | |
| ignore_data_skip=False, | |
| include_inputs_for_metrics=False, | |
| include_num_input_tokens_seen=False, | |
| include_tokens_per_second=False, | |
| jit_mode_eval=False, | |
| label_names=None, | |
| label_smoothing_factor=0.0, | |
| learning_rate=2e-05, | |
| length_column_name=length, | |
| load_best_model_at_end=False, | |
| local_rank=2, | |
| log_level=passive, | |
| log_level_replica=warning, | |
| log_on_each_node=True, | |
| logging_dir=/home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909/runs/Mar26_12-48-05_siat-a100-4-01, | |
| logging_first_step=False, | |
| logging_nan_inf_filter=True, | |
| logging_steps=10, | |
| logging_strategy=steps, | |
| lr_scheduler_kwargs={}, | |
| lr_scheduler_type=cosine, | |
| max_grad_norm=1.0, | |
| max_steps=-1, | |
| metric_for_best_model=None, | |
| mp_parameters=, | |
| neftune_noise_alpha=None, | |
| no_cuda=False, | |
| num_train_epochs=3.0, | |
| optim=adamw_torch, | |
| optim_args=None, | |
| output_dir=/home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909, | |
| overwrite_output_dir=True, | |
| past_index=-1, | |
| per_device_eval_batch_size=8, | |
| per_device_train_batch_size=32, | |
| predict_with_generate=False, | |
| prediction_loss_only=False, | |
| push_to_hub=False, | |
| push_to_hub_model_id=None, | |
| push_to_hub_organization=None, | |
| push_to_hub_token=<PUSH_TO_HUB_TOKEN>, | |
| ray_scope=last, | |
| remove_unused_columns=True, | |
| report_to=[], | |
| resume_from_checkpoint=None, | |
| run_name=/home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909, | |
| save_on_each_node=False, | |
| save_only_model=False, | |
| save_safetensors=True, | |
| save_steps=1000, | |
| save_strategy=steps, | |
| save_total_limit=None, | |
| seed=42, | |
| skip_memory_metrics=True, | |
| sortish_sampler=False, | |
| split_batches=None, | |
| tf32=None, | |
| torch_compile=False, | |
| torch_compile_backend=None, | |
| torch_compile_mode=None, | |
| torchdynamo=None, | |
| tpu_metrics_debug=False, | |
| tpu_num_cores=None, | |
| use_cpu=False, | |
| use_ipex=False, | |
| use_legacy_prediction_loop=False, | |
| use_mps_device=False, | |
| warmup_ratio=0.1, | |
| warmup_steps=0, | |
| weight_decay=0.0, | |
| ) | |
| You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 | |
| You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 | |
| Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]/home/lirenhao/anaconda3/envs/llama_factory/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() | |
| return self.fget.__get__(instance, owner)() | |
| Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]/home/lirenhao/anaconda3/envs/llama_factory/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() | |
| return self.fget.__get__(instance, owner)() | |
| Loading checkpoint shards: 50%|█████ | 1/2 [00:00<00:00, 5.73it/s] Loading checkpoint shards: 50%|█████ | 1/2 [00:00<00:00, 5.53it/s] Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 6.11it/s] Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 6.04it/s] | |
| 03/26/2024 12:48:05 - INFO - llmtuner.model.patcher - Gradient checkpointing enabled. | |
| 03/26/2024 12:48:05 - INFO - llmtuner.model.adapter - Fine-tuning method: Full | |
| Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 5.91it/s] Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 5.85it/s] | |
| 03/26/2024 12:48:05 - INFO - llmtuner.model.patcher - Gradient checkpointing enabled. | |
| 03/26/2024 12:48:05 - INFO - llmtuner.model.adapter - Fine-tuning method: Full | |
| 03/26/2024 12:48:06 - INFO - llmtuner.hparams.parser - Process rank: 0, device: cuda:0, n_gpu: 1 | |
| distributed training: True, compute dtype: torch.bfloat16 | |
| 03/26/2024 12:48:06 - INFO - llmtuner.hparams.parser - Training/evaluation parameters Seq2SeqTrainingArguments( | |
| _n_gpu=1, | |
| accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True}, | |
| adafactor=False, | |
| adam_beta1=0.9, | |
| adam_beta2=0.999, | |
| adam_epsilon=1e-08, | |
| auto_find_batch_size=False, | |
| bf16=True, | |
| bf16_full_eval=False, | |
| data_seed=None, | |
| dataloader_drop_last=False, | |
| dataloader_num_workers=0, | |
| dataloader_persistent_workers=False, | |
| dataloader_pin_memory=True, | |
| dataloader_prefetch_factor=None, | |
| ddp_backend=None, | |
| ddp_broadcast_buffers=None, | |
| ddp_bucket_cap_mb=None, | |
| ddp_find_unused_parameters=None, | |
| ddp_timeout=1800, | |
| debug=[], | |
| deepspeed=/home/lirenhao/projects/LLaMA-Factory/ds_configs/zero2_no_offload.json, | |
| disable_tqdm=False, | |
| dispatch_batches=None, | |
| do_eval=False, | |
| do_predict=False, | |
| do_train=True, | |
| eval_accumulation_steps=None, | |
| eval_delay=0, | |
| eval_steps=None, | |
| evaluation_strategy=no, | |
| fp16=False, | |
| fp16_backend=auto, | |
| fp16_full_eval=False, | |
| fp16_opt_level=O1, | |
| fsdp=[], | |
| fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, | |
| fsdp_min_num_params=0, | |
| fsdp_transformer_layer_cls_to_wrap=None, | |
| full_determinism=False, | |
| generation_config=None, | |
| generation_max_length=None, | |
| generation_num_beams=None, | |
| gradient_accumulation_steps=4, | |
| gradient_checkpointing=False, | |
| gradient_checkpointing_kwargs=None, | |
| greater_is_better=None, | |
| group_by_length=False, | |
| half_precision_backend=auto, | |
| hub_always_push=False, | |
| hub_model_id=None, | |
| hub_private_repo=False, | |
| hub_strategy=every_save, | |
| hub_token=<HUB_TOKEN>, | |
| ignore_data_skip=False, | |
| include_inputs_for_metrics=False, | |
| include_num_input_tokens_seen=False, | |
| include_tokens_per_second=False, | |
| jit_mode_eval=False, | |
| label_names=None, | |
| label_smoothing_factor=0.0, | |
| learning_rate=2e-05, | |
| length_column_name=length, | |
| load_best_model_at_end=False, | |
| local_rank=0, | |
| log_level=passive, | |
| log_level_replica=warning, | |
| log_on_each_node=True, | |
| logging_dir=/home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909/runs/Mar26_12-48-05_siat-a100-4-01, | |
| logging_first_step=False, | |
| logging_nan_inf_filter=True, | |
| logging_steps=10, | |
| logging_strategy=steps, | |
| lr_scheduler_kwargs={}, | |
| lr_scheduler_type=cosine, | |
| max_grad_norm=1.0, | |
| max_steps=-1, | |
| metric_for_best_model=None, | |
| mp_parameters=, | |
| neftune_noise_alpha=None, | |
| no_cuda=False, | |
| num_train_epochs=3.0, | |
| optim=adamw_torch, | |
| optim_args=None, | |
| output_dir=/home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909, | |
| overwrite_output_dir=True, | |
| past_index=-1, | |
| per_device_eval_batch_size=8, | |
| per_device_train_batch_size=32, | |
| predict_with_generate=False, | |
| prediction_loss_only=False, | |
| push_to_hub=False, | |
| push_to_hub_model_id=None, | |
| push_to_hub_organization=None, | |
| push_to_hub_token=<PUSH_TO_HUB_TOKEN>, | |
| ray_scope=last, | |
| remove_unused_columns=True, | |
| report_to=[], | |
| resume_from_checkpoint=None, | |
| run_name=/home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909, | |
| save_on_each_node=False, | |
| save_only_model=False, | |
| save_safetensors=True, | |
| save_steps=1000, | |
| save_strategy=steps, | |
| save_total_limit=None, | |
| seed=42, | |
| skip_memory_metrics=True, | |
| sortish_sampler=False, | |
| split_batches=None, | |
| tf32=None, | |
| torch_compile=False, | |
| torch_compile_backend=None, | |
| torch_compile_mode=None, | |
| torchdynamo=None, | |
| tpu_metrics_debug=False, | |
| tpu_num_cores=None, | |
| use_cpu=False, | |
| use_ipex=False, | |
| use_legacy_prediction_loop=False, | |
| use_mps_device=False, | |
| warmup_ratio=0.1, | |
| warmup_steps=0, | |
| weight_decay=0.0, | |
| ) | |
| [INFO|tokenization_utils_base.py:2044] 2024-03-26 12:48:06,154 >> loading file tokenizer.model | |
| [INFO|tokenization_utils_base.py:2044] 2024-03-26 12:48:06,154 >> loading file added_tokens.json | |
| [INFO|tokenization_utils_base.py:2044] 2024-03-26 12:48:06,154 >> loading file special_tokens_map.json | |
| [INFO|tokenization_utils_base.py:2044] 2024-03-26 12:48:06,154 >> loading file tokenizer_config.json | |
| [INFO|tokenization_utils_base.py:2044] 2024-03-26 12:48:06,154 >> loading file tokenizer.json | |
| [WARNING|logging.py:329] 2024-03-26 12:48:06,212 >> You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 | |
| [INFO|configuration_utils.py:726] 2024-03-26 12:48:06,293 >> loading configuration file /home/lirenhao/pretrained_models/Llama-2-7b-hf/config.json | |
| [INFO|configuration_utils.py:791] 2024-03-26 12:48:06,294 >> Model config LlamaConfig { | |
| "_name_or_path": "/home/lirenhao/pretrained_models/Llama-2-7b-hf", | |
| "architectures": [ | |
| "LlamaForCausalLM" | |
| ], | |
| "attention_bias": false, | |
| "attention_dropout": 0.0, | |
| "bos_token_id": 1, | |
| "eos_token_id": 2, | |
| "hidden_act": "silu", | |
| "hidden_size": 4096, | |
| "initializer_range": 0.02, | |
| "intermediate_size": 11008, | |
| "max_position_embeddings": 2048, | |
| "model_type": "llama", | |
| "num_attention_heads": 32, | |
| "num_hidden_layers": 32, | |
| "num_key_value_heads": 32, | |
| "pretraining_tp": 1, | |
| "rms_norm_eps": 1e-05, | |
| "rope_scaling": null, | |
| "rope_theta": 10000.0, | |
| "tie_word_embeddings": false, | |
| "torch_dtype": "bfloat16", | |
| "transformers_version": "4.38.1", | |
| "use_cache": true, | |
| "vocab_size": 32000 | |
| } | |
| [INFO|modeling_utils.py:3254] 2024-03-26 12:48:06,320 >> loading weights file /home/lirenhao/pretrained_models/Llama-2-7b-hf/pytorch_model.bin.index.json | |
| [INFO|modeling_utils.py:1400] 2024-03-26 12:48:06,321 >> Instantiating LlamaForCausalLM model under default dtype torch.bfloat16. | |
| 03/26/2024 12:48:06 - INFO - llmtuner.hparams.parser - Process rank: 3, device: cuda:3, n_gpu: 1 | |
| distributed training: True, compute dtype: torch.bfloat16 | |
| [INFO|configuration_utils.py:845] 2024-03-26 12:48:06,322 >> Generate config GenerationConfig { | |
| "bos_token_id": 1, | |
| "eos_token_id": 2 | |
| } | |
| 03/26/2024 12:48:06 - INFO - llmtuner.hparams.parser - Training/evaluation parameters Seq2SeqTrainingArguments( | |
| _n_gpu=1, | |
| accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True}, | |
| adafactor=False, | |
| adam_beta1=0.9, | |
| adam_beta2=0.999, | |
| adam_epsilon=1e-08, | |
| auto_find_batch_size=False, | |
| bf16=True, | |
| bf16_full_eval=False, | |
| data_seed=None, | |
| dataloader_drop_last=False, | |
| dataloader_num_workers=0, | |
| dataloader_persistent_workers=False, | |
| dataloader_pin_memory=True, | |
| dataloader_prefetch_factor=None, | |
| ddp_backend=None, | |
| ddp_broadcast_buffers=None, | |
| ddp_bucket_cap_mb=None, | |
| ddp_find_unused_parameters=None, | |
| ddp_timeout=1800, | |
| debug=[], | |
| deepspeed=/home/lirenhao/projects/LLaMA-Factory/ds_configs/zero2_no_offload.json, | |
| disable_tqdm=False, | |
| dispatch_batches=None, | |
| do_eval=False, | |
| do_predict=False, | |
| do_train=True, | |
| eval_accumulation_steps=None, | |
| eval_delay=0, | |
| eval_steps=None, | |
| evaluation_strategy=no, | |
| fp16=False, | |
| fp16_backend=auto, | |
| fp16_full_eval=False, | |
| fp16_opt_level=O1, | |
| fsdp=[], | |
| fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}, | |
| fsdp_min_num_params=0, | |
| fsdp_transformer_layer_cls_to_wrap=None, | |
| full_determinism=False, | |
| generation_config=None, | |
| generation_max_length=None, | |
| generation_num_beams=None, | |
| gradient_accumulation_steps=4, | |
| gradient_checkpointing=False, | |
| gradient_checkpointing_kwargs=None, | |
| greater_is_better=None, | |
| group_by_length=False, | |
| half_precision_backend=auto, | |
| hub_always_push=False, | |
| hub_model_id=None, | |
| hub_private_repo=False, | |
| hub_strategy=every_save, | |
| hub_token=<HUB_TOKEN>, | |
| ignore_data_skip=False, | |
| include_inputs_for_metrics=False, | |
| include_num_input_tokens_seen=False, | |
| include_tokens_per_second=False, | |
| jit_mode_eval=False, | |
| label_names=None, | |
| label_smoothing_factor=0.0, | |
| learning_rate=2e-05, | |
| length_column_name=length, | |
| load_best_model_at_end=False, | |
| local_rank=3, | |
| log_level=passive, | |
| log_level_replica=warning, | |
| log_on_each_node=True, | |
| logging_dir=/home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909/runs/Mar26_12-48-05_siat-a100-4-01, | |
| logging_first_step=False, | |
| logging_nan_inf_filter=True, | |
| logging_steps=10, | |
| logging_strategy=steps, | |
| lr_scheduler_kwargs={}, | |
| lr_scheduler_type=cosine, | |
| max_grad_norm=1.0, | |
| max_steps=-1, | |
| metric_for_best_model=None, | |
| mp_parameters=, | |
| neftune_noise_alpha=None, | |
| no_cuda=False, | |
| num_train_epochs=3.0, | |
| optim=adamw_torch, | |
| optim_args=None, | |
| output_dir=/home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909, | |
| overwrite_output_dir=True, | |
| past_index=-1, | |
| per_device_eval_batch_size=8, | |
| per_device_train_batch_size=32, | |
| predict_with_generate=False, | |
| prediction_loss_only=False, | |
| push_to_hub=False, | |
| push_to_hub_model_id=None, | |
| push_to_hub_organization=None, | |
| push_to_hub_token=<PUSH_TO_HUB_TOKEN>, | |
| ray_scope=last, | |
| remove_unused_columns=True, | |
| report_to=[], | |
| resume_from_checkpoint=None, | |
| run_name=/home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909, | |
| save_on_each_node=False, | |
| save_only_model=False, | |
| save_safetensors=True, | |
| save_steps=1000, | |
| save_strategy=steps, | |
| save_total_limit=None, | |
| seed=42, | |
| skip_memory_metrics=True, | |
| sortish_sampler=False, | |
| split_batches=None, | |
| tf32=None, | |
| torch_compile=False, | |
| torch_compile_backend=None, | |
| torch_compile_mode=None, | |
| torchdynamo=None, | |
| tpu_metrics_debug=False, | |
| tpu_num_cores=None, | |
| use_cpu=False, | |
| use_ipex=False, | |
| use_legacy_prediction_loop=False, | |
| use_mps_device=False, | |
| warmup_ratio=0.1, | |
| warmup_steps=0, | |
| weight_decay=0.0, | |
| ) | |
| You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 | |
| Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]/home/lirenhao/anaconda3/envs/llama_factory/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() | |
| return self.fget.__get__(instance, owner)() | |
| Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]/home/lirenhao/anaconda3/envs/llama_factory/lib/python3.10/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() | |
| return self.fget.__get__(instance, owner)() | |
| Loading checkpoint shards: 50%|█████ | 1/2 [00:00<00:00, 3.41it/s] Loading checkpoint shards: 50%|█████ | 1/2 [00:00<00:00, 3.23it/s] Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 3.61it/s] Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 3.58it/s] | |
| [INFO|modeling_utils.py:3992] 2024-03-26 12:48:07,021 >> All model checkpoint weights were used when initializing LlamaForCausalLM. | |
| [INFO|modeling_utils.py:4000] 2024-03-26 12:48:07,021 >> All the weights of LlamaForCausalLM were initialized from the model checkpoint at /home/lirenhao/pretrained_models/Llama-2-7b-hf. | |
| If your task is similar to the task the model of the checkpoint was trained on, you can already use LlamaForCausalLM for predictions without further training. | |
| [INFO|configuration_utils.py:798] 2024-03-26 12:48:07,024 >> loading configuration file /home/lirenhao/pretrained_models/Llama-2-7b-hf/generation_config.json | |
| [INFO|configuration_utils.py:845] 2024-03-26 12:48:07,025 >> Generate config GenerationConfig { | |
| "bos_token_id": 1, | |
| "eos_token_id": 2 | |
| } | |
| 03/26/2024 12:48:07 - INFO - llmtuner.model.patcher - Gradient checkpointing enabled. | |
| 03/26/2024 12:48:07 - INFO - llmtuner.model.adapter - Fine-tuning method: Full | |
| Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 3.51it/s] Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 3.46it/s] | |
| 03/26/2024 12:48:07 - INFO - llmtuner.model.patcher - Gradient checkpointing enabled. | |
| 03/26/2024 12:48:07 - INFO - llmtuner.model.adapter - Fine-tuning method: Full | |
| 03/26/2024 12:48:08 - INFO - llmtuner.model.loader - trainable params: 6738415616 || all params: 6738415616 || trainable%: 100.0000 | |
| 03/26/2024 12:48:08 - INFO - llmtuner.data.template - Add pad token: </s> | |
| 03/26/2024 12:48:10 - INFO - llmtuner.model.loader - trainable params: 6738415616 || all params: 6738415616 || trainable%: 100.0000 | |
| 03/26/2024 12:48:10 - INFO - llmtuner.data.template - Add pad token: </s> | |
| 03/26/2024 12:48:10 - INFO - llmtuner.model.loader - trainable params: 6738415616 || all params: 6738415616 || trainable%: 100.0000 | |
| 03/26/2024 12:48:10 - INFO - llmtuner.data.template - Add pad token: </s> | |
| 03/26/2024 12:48:10 - INFO - llmtuner.data.loader - Loading dataset /home/lirenhao/projects/ContrastEvol/data/sft/alpaca_random9k_evol-chatgpt.json... | |
| 03/26/2024 12:48:10 - WARNING - llmtuner.data.utils - Checksum failed: missing SHA-1 hash value in dataset_info.json. | |
| 03/26/2024 12:48:10 - INFO - llmtuner.model.loader - trainable params: 6738415616 || all params: 6738415616 || trainable%: 100.0000 | |
| 03/26/2024 12:48:10 - INFO - llmtuner.data.template - Add pad token: </s> | |
| Using custom data configuration default-414a836d16729ac4 | |
| Loading Dataset Infos from /home/lirenhao/anaconda3/envs/llama_factory/lib/python3.10/site-packages/datasets/packaged_modules/json | |
| Overwrite dataset info from restored data version if exists. | |
| Loading Dataset info from /home/lirenhao/.cache/huggingface/datasets/json/default-414a836d16729ac4/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 | |
| Found cached dataset json (/home/lirenhao/.cache/huggingface/datasets/json/default-414a836d16729ac4/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96) | |
| Loading Dataset info from /home/lirenhao/.cache/huggingface/datasets/json/default-414a836d16729ac4/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96 | |
| Converting format of dataset: 0%| | 0/9229 [00:00<?, ? examples/s]Caching processed dataset at /home/lirenhao/.cache/huggingface/datasets/json/default-414a836d16729ac4/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-1daed2d67ac7782d.arrow | |
| Converting format of dataset: 100%|██████████| 9229/9229 [00:00<00:00, 88514.05 examples/s] Converting format of dataset: 100%|██████████| 9229/9229 [00:00<00:00, 86377.54 examples/s] | |
| 03/26/2024 12:48:13 - INFO - llmtuner.data.loader - Loading dataset /home/lirenhao/projects/ContrastEvol/data/sft/alpaca_random9k_evol-chatgpt.json... | |
| 03/26/2024 12:48:13 - INFO - llmtuner.data.loader - Loading dataset /home/lirenhao/projects/ContrastEvol/data/sft/alpaca_random9k_evol-chatgpt.json... | |
| 03/26/2024 12:48:13 - INFO - llmtuner.data.loader - Loading dataset /home/lirenhao/projects/ContrastEvol/data/sft/alpaca_random9k_evol-chatgpt.json... | |
| 03/26/2024 12:48:13 - WARNING - llmtuner.data.utils - Checksum failed: missing SHA-1 hash value in dataset_info.json. | |
| 03/26/2024 12:48:13 - WARNING - llmtuner.data.utils - Checksum failed: missing SHA-1 hash value in dataset_info.json. | |
| 03/26/2024 12:48:13 - WARNING - llmtuner.data.utils - Checksum failed: missing SHA-1 hash value in dataset_info.json. | |
| Running tokenizer on dataset: 0%| | 0/9229 [00:00<?, ? examples/s] Converting format of dataset: 0%| | 0/9229 [00:00<?, ? examples/s] Converting format of dataset: 0%| | 0/9229 [00:00<?, ? examples/s] Converting format of dataset: 0%| | 0/9229 [00:00<?, ? examples/s] Converting format of dataset: 100%|██████████| 9229/9229 [00:00<00:00, 67545.95 examples/s] Converting format of dataset: 100%|██████████| 9229/9229 [00:00<00:00, 52426.46 examples/s] Converting format of dataset: 100%|██████████| 9229/9229 [00:00<00:00, 66672.00 examples/s] | |
| Converting format of dataset: 100%|██████████| 9229/9229 [00:00<00:00, 35383.15 examples/s] Converting format of dataset: 100%|██████████| 9229/9229 [00:00<00:00, 28475.48 examples/s] | |
| Converting format of dataset: 100%|██████████| 9229/9229 [00:00<00:00, 35137.25 examples/s] | |
| Caching processed dataset at /home/lirenhao/.cache/huggingface/datasets/json/default-414a836d16729ac4/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96/cache-d2ba94cd255b6779.arrow | |
| Running tokenizer on dataset: 11%|█ | 1000/9229 [00:01<00:14, 583.55 examples/s] Running tokenizer on dataset: 22%|██▏ | 2000/9229 [00:03<00:12, 578.47 examples/s] Running tokenizer on dataset: 33%|███▎ | 3000/9229 [00:05<00:10, 581.57 examples/s] Running tokenizer on dataset: 43%|████▎ | 4000/9229 [00:06<00:09, 579.86 examples/s] Running tokenizer on dataset: 54%|█████▍ | 5000/9229 [00:08<00:07, 580.23 examples/s] Running tokenizer on dataset: 65%|██████▌ | 6000/9229 [00:10<00:05, 584.08 examples/s] Running tokenizer on dataset: 76%|███████▌ | 7000/9229 [00:11<00:03, 586.29 examples/s] Running tokenizer on dataset: 87%|████████▋ | 8000/9229 [00:13<00:02, 591.12 examples/s] Running tokenizer on dataset: 98%|█████████▊| 9000/9229 [00:15<00:00, 583.21 examples/s] Running tokenizer on dataset: 100%|██████████| 9229/9229 [00:15<00:00, 588.39 examples/s] Running tokenizer on dataset: 100%|██████████| 9229/9229 [00:16<00:00, 563.74 examples/s] | |
| input_ids: | |
| [1, 518, 25580, 29962, 3532, 14816, 29903, 6778, 13, 3492, 526, 263, 8444, 29892, 3390, 1319, 322, 15993, 20255, 29889, 29849, 1234, 408, 1371, 3730, 408, 1950, 29892, 1550, 1641, 9109, 29889, 3575, 6089, 881, 451, 3160, 738, 10311, 1319, 29892, 443, 621, 936, 29892, 11021, 391, 29892, 7916, 391, 29892, 304, 27375, 29892, 18215, 29892, 470, 27302, 2793, 29889, 3529, 9801, 393, 596, 20890, 526, 5374, 635, 443, 5365, 1463, 322, 6374, 297, 5469, 29889, 13, 13, 3644, 263, 1139, 947, 451, 1207, 738, 4060, 29892, 470, 338, 451, 2114, 1474, 16165, 261, 296, 29892, 5649, 2020, 2012, 310, 22862, 1554, 451, 1959, 29889, 960, 366, 1016, 29915, 29873, 1073, 278, 1234, 304, 263, 1139, 29892, 3113, 1016, 29915, 29873, 6232, 2089, 2472, 29889, 13, 29966, 829, 14816, 29903, 6778, 13, 13, 11139, 3034, 675, 278, 26627, 1199, 310, 2155, 2296, 261, 273, 29915, 29879, 4823, 525, 24111, 310, 887, 29915, 29973, 518, 29914, 25580, 29962, 450, 4823, 525, 24111, 310, 887, 29915, 491, 2155, 2296, 261, 273, 2011, 764, 29879, 278, 4327, 1230, 3081, 310, 5360, 373, 278, 15572, 391, 29915, 29879, 6030, 29889, 450, 26627, 1199, 28475, 920, 278, 15572, 391, 29915, 29879, 5683, 29899, 11147, 760, 5414, 322, 3209, 950, 1652, 381, 1259, 2041, 304, 385, 1095, 408, 540, 20074, 297, 5360, 29892, 12141, 292, 278, 10879, 310, 5360, 373, 670, 301, 7004, 1508, 29889, 1152, 2777, 29892, 278, 26627, 1199, 376, 1576, 4402, 3508, 29915, 29873, 278, 1900, 2058, 304, 1284, 263, 658, 369, 847, 1105, 278, 2594, 338, 988, 306, 748, 29908, 12266, 263, 9500, 297, 6030, 2861, 304, 278, 9949, 310, 5360, 29889, 19814, 29892, 278, 521, 16566, 19310, 7093, 278, 2600, 618, 2779, 310, 5360, 29892, 26233, 393, 372, 756, 278, 11509, 304, 8267, 322, 1735, 385, 5375, 29889, 910, 2011, 764, 284, 8128, 263, 4948, 8362, 8004, 310, 278, 4823, 29915, 29879, 2643, 29892, 1510, 29883, 5832, 278, 15572, 391, 29915, 29879, 7333, 14321, 322, 13852, 1549, 5360, 29889, 2] | |
| inputs: | |
| <s> [INST] <<SYS>> | |
| You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. | |
| If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information. | |
| <</SYS>> | |
| Summarize the lyrics of Ed Sheeran's song 'Shape of You'? [/INST] The song 'Shape of You' by Ed Sheeran portrays the transformative power of love on the protagonist's behavior. The lyrics illustrate how the protagonist's late-night partying and casual flirting come to an end as he falls in love, highlighting the impact of love on his lifestyle. For instance, the lyrics "The club isn't the best place to find a lover / So the bar is where I go" indicate a shift in behavior due to the influence of love. Additionally, the chorus emphasizes the profound effect of love, suggesting that it has the ability to shape and change an individual. This portrayal provides a nuanced understanding of the song's message, showcasing the protagonist's personal growth and transformation through love.</s> | |
| label_ids: | |
| [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 450, 4823, 525, 24111, 310, 887, 29915, 491, 2155, 2296, 261, 273, 2011, 764, 29879, 278, 4327, 1230, 3081, 310, 5360, 373, 278, 15572, 391, 29915, 29879, 6030, 29889, 450, 26627, 1199, 28475, 920, 278, 15572, 391, 29915, 29879, 5683, 29899, 11147, 760, 5414, 322, 3209, 950, 1652, 381, 1259, 2041, 304, 385, 1095, 408, 540, 20074, 297, 5360, 29892, 12141, 292, 278, 10879, 310, 5360, 373, 670, 301, 7004, 1508, 29889, 1152, 2777, 29892, 278, 26627, 1199, 376, 1576, 4402, 3508, 29915, 29873, 278, 1900, 2058, 304, 1284, 263, 658, 369, 847, 1105, 278, 2594, 338, 988, 306, 748, 29908, 12266, 263, 9500, 297, 6030, 2861, 304, 278, 9949, 310, 5360, 29889, 19814, 29892, 278, 521, 16566, 19310, 7093, 278, 2600, 618, 2779, 310, 5360, 29892, 26233, 393, 372, 756, 278, 11509, 304, 8267, 322, 1735, 385, 5375, 29889, 910, 2011, 764, 284, 8128, 263, 4948, 8362, 8004, 310, 278, 4823, 29915, 29879, 2643, 29892, 1510, 29883, 5832, 278, 15572, 391, 29915, 29879, 7333, 14321, 322, 13852, 1549, 5360, 29889, 2] | |
| labels: | |
| The song 'Shape of You' by Ed Sheeran portrays the transformative power of love on the protagonist's behavior. The lyrics illustrate how the protagonist's late-night partying and casual flirting come to an end as he falls in love, highlighting the impact of love on his lifestyle. For instance, the lyrics "The club isn't the best place to find a lover / So the bar is where I go" indicate a shift in behavior due to the influence of love. Additionally, the chorus emphasizes the profound effect of love, suggesting that it has the ability to shape and change an individual. This portrayal provides a nuanced understanding of the song's message, showcasing the protagonist's personal growth and transformation through love.</s> | |
| [INFO|training_args.py:1902] 2024-03-26 12:48:29,406 >> PyTorch: setting up devices | |
| Running tokenizer on dataset: 0%| | 0/9229 [00:00<?, ? examples/s] Running tokenizer on dataset: 0%| | 0/9229 [00:00<?, ? examples/s] Running tokenizer on dataset: 0%| | 0/9229 [00:00<?, ? examples/s]/home/lirenhao/anaconda3/envs/llama_factory/lib/python3.10/site-packages/transformers/training_args.py:1815: FutureWarning: `--push_to_hub_token` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_token` instead. | |
| warnings.warn( | |
| Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. | |
| [INFO|trainer.py:601] 2024-03-26 12:48:29,456 >> Using auto half precision backend | |
| [2024-03-26 12:48:29,609] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.13.1, git-hash=unknown, git-branch=unknown | |
| Running tokenizer on dataset: 11%|█ | 1000/9229 [00:01<00:14, 563.11 examples/s] Running tokenizer on dataset: 11%|█ | 1000/9229 [00:01<00:14, 559.57 examples/s] Running tokenizer on dataset: 11%|█ | 1000/9229 [00:01<00:14, 552.50 examples/s] Running tokenizer on dataset: 22%|██▏ | 2000/9229 [00:03<00:12, 559.96 examples/s] Running tokenizer on dataset: 22%|██▏ | 2000/9229 [00:03<00:13, 555.17 examples/s] Running tokenizer on dataset: 22%|██▏ | 2000/9229 [00:03<00:13, 555.39 examples/s] Running tokenizer on dataset: 33%|███▎ | 3000/9229 [00:05<00:11, 560.36 examples/s] Running tokenizer on dataset: 33%|███▎ | 3000/9229 [00:05<00:11, 561.99 examples/s] Running tokenizer on dataset: 33%|███▎ | 3000/9229 [00:05<00:11, 562.93 examples/s] Running tokenizer on dataset: 43%|████▎ | 4000/9229 [00:07<00:09, 563.82 examples/s] Running tokenizer on dataset: 43%|████▎ | 4000/9229 [00:07<00:09, 562.85 examples/s] Running tokenizer on dataset: 43%|████▎ | 4000/9229 [00:07<00:09, 561.71 examples/s] Running tokenizer on dataset: 54%|█████▍ | 5000/9229 [00:08<00:07, 565.95 examples/s] Running tokenizer on dataset: 54%|█████▍ | 5000/9229 [00:08<00:07, 564.55 examples/s] Running tokenizer on dataset: 54%|█████▍ | 5000/9229 [00:08<00:07, 563.17 examples/s] Running tokenizer on dataset: 65%|██████▌ | 6000/9229 [00:10<00:05, 571.07 examples/s] Running tokenizer on dataset: 65%|██████▌ | 6000/9229 [00:10<00:05, 568.90 examples/s] Running tokenizer on dataset: 65%|██████▌ | 6000/9229 [00:10<00:05, 567.98 examples/s] Running tokenizer on dataset: 76%|███████▌ | 7000/9229 [00:12<00:03, 573.86 examples/s] Running tokenizer on dataset: 76%|███████▌ | 7000/9229 [00:12<00:03, 571.14 examples/s] Running tokenizer on dataset: 76%|███████▌ | 7000/9229 [00:12<00:03, 570.29 examples/s] Running tokenizer on dataset: 87%|████████▋ | 8000/9229 [00:14<00:02, 578.13 examples/s] Running tokenizer on dataset: 87%|████████▋ | 8000/9229 [00:14<00:02, 574.89 examples/s] Running tokenizer on dataset: 87%|████████▋ | 8000/9229 [00:14<00:02, 575.09 examples/s] Running tokenizer on dataset: 98%|█████████▊| 9000/9229 [00:15<00:00, 577.65 examples/s] Running tokenizer on dataset: 98%|█████████▊| 9000/9229 [00:15<00:00, 574.85 examples/s] Running tokenizer on dataset: 98%|█████████▊| 9000/9229 [00:15<00:00, 574.08 examples/s] Running tokenizer on dataset: 100%|██████████| 9229/9229 [00:16<00:00, 581.68 examples/s] Running tokenizer on dataset: 100%|██████████| 9229/9229 [00:16<00:00, 579.13 examples/s] Running tokenizer on dataset: 100%|██████████| 9229/9229 [00:16<00:00, 578.23 examples/s] Running tokenizer on dataset: 100%|██████████| 9229/9229 [00:17<00:00, 542.16 examples/s] | |
| /home/lirenhao/anaconda3/envs/llama_factory/lib/python3.10/site-packages/transformers/training_args.py:1815: FutureWarning: `--push_to_hub_token` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_token` instead. | |
| warnings.warn( | |
| Running tokenizer on dataset: 100%|██████████| 9229/9229 [00:17<00:00, 534.58 examples/s] | |
| Running tokenizer on dataset: 100%|██████████| 9229/9229 [00:17<00:00, 534.55 examples/s] | |
| /home/lirenhao/anaconda3/envs/llama_factory/lib/python3.10/site-packages/transformers/training_args.py:1815: FutureWarning: `--push_to_hub_token` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_token` instead. | |
| warnings.warn( | |
| /home/lirenhao/anaconda3/envs/llama_factory/lib/python3.10/site-packages/transformers/training_args.py:1815: FutureWarning: `--push_to_hub_token` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_token` instead. | |
| warnings.warn( | |
| [2024-03-26 12:48:53,454] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False | |
| Using /home/lirenhao/.cache/torch_extensions/py310_cu121 as PyTorch extensions root... | |
| Using /home/lirenhao/.cache/torch_extensions/py310_cu121 as PyTorch extensions root...Using /home/lirenhao/.cache/torch_extensions/py310_cu121 as PyTorch extensions root... | |
| Using /home/lirenhao/.cache/torch_extensions/py310_cu121 as PyTorch extensions root... | |
| Detected CUDA files, patching ldflags | |
| Emitting ninja build file /home/lirenhao/.cache/torch_extensions/py310_cu121/fused_adam/build.ninja... | |
| Building extension module fused_adam... | |
| Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) | |
| ninja: no work to do. | |
| Loading extension module fused_adam... | |
| Time to load fused_adam op: 0.057227373123168945 seconds | |
| /home/lirenhao/anaconda3/envs/llama_factory/lib/python3.10/site-packages/deepspeed/ops/adam/fused_adam.py:96: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.) | |
| self._dummy_overflow_buf = get_accelerator().IntTensor([0]) | |
| Loading extension module fused_adam...Loading extension module fused_adam... | |
| Time to load fused_adam op: 0.10106301307678223 secondsTime to load fused_adam op: 0.10106754302978516 seconds | |
| Loading extension module fused_adam... | |
| /home/lirenhao/anaconda3/envs/llama_factory/lib/python3.10/site-packages/deepspeed/ops/adam/fused_adam.py:96: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.) | |
| self._dummy_overflow_buf = get_accelerator().IntTensor([0]) | |
| /home/lirenhao/anaconda3/envs/llama_factory/lib/python3.10/site-packages/deepspeed/ops/adam/fused_adam.py:96: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.) | |
| self._dummy_overflow_buf = get_accelerator().IntTensor([0]) | |
| [2024-03-26 12:48:53,722] [INFO] [logging.py:96:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer | |
| [2024-03-26 12:48:53,722] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer | |
| Time to load fused_adam op: 0.10103702545166016 seconds | |
| /home/lirenhao/anaconda3/envs/llama_factory/lib/python3.10/site-packages/deepspeed/ops/adam/fused_adam.py:96: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.) | |
| self._dummy_overflow_buf = get_accelerator().IntTensor([0]) | |
| [2024-03-26 12:48:53,733] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam | |
| [2024-03-26 12:48:53,733] [INFO] [utils.py:56:is_zero_supported_optimizer] Checking ZeRO support for optimizer=FusedAdam type=<class 'deepspeed.ops.adam.fused_adam.FusedAdam'> | |
| [2024-03-26 12:48:53,733] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 2 optimizer | |
| [2024-03-26 12:48:53,733] [INFO] [stage_1_and_2.py:143:__init__] Reduce bucket size 200000000 | |
| [2024-03-26 12:48:53,733] [INFO] [stage_1_and_2.py:144:__init__] Allgather bucket size 200000000 | |
| [2024-03-26 12:48:53,733] [INFO] [stage_1_and_2.py:145:__init__] CPU Offload: False | |
| [2024-03-26 12:48:53,733] [INFO] [stage_1_and_2.py:146:__init__] Round robin gradient partitioning: False | |
| [2024-03-26 12:49:07,437] [INFO] [utils.py:791:see_memory_usage] Before initializing optimizer states | |
| [2024-03-26 12:49:07,438] [INFO] [utils.py:792:see_memory_usage] MA 18.86 GB Max_MA 22.0 GB CA 22.0 GB Max_CA 22 GB | |
| [2024-03-26 12:49:07,438] [INFO] [utils.py:799:see_memory_usage] CPU Virtual Memory: used = 95.71 GB, percent = 9.9% | |
| [2024-03-26 12:49:07,626] [INFO] [utils.py:791:see_memory_usage] After initializing optimizer states | |
| [2024-03-26 12:49:07,627] [INFO] [utils.py:792:see_memory_usage] MA 31.41 GB Max_MA 37.69 GB CA 40.83 GB Max_CA 41 GB | |
| [2024-03-26 12:49:07,627] [INFO] [utils.py:799:see_memory_usage] CPU Virtual Memory: used = 87.46 GB, percent = 9.1% | |
| [2024-03-26 12:49:07,627] [INFO] [stage_1_and_2.py:533:__init__] optimizer state initialized | |
| [2024-03-26 12:49:07,786] [INFO] [utils.py:791:see_memory_usage] After initializing ZeRO optimizer | |
| [2024-03-26 12:49:07,787] [INFO] [utils.py:792:see_memory_usage] MA 31.41 GB Max_MA 31.41 GB CA 40.83 GB Max_CA 41 GB | |
| [2024-03-26 12:49:07,787] [INFO] [utils.py:799:see_memory_usage] CPU Virtual Memory: used = 75.46 GB, percent = 7.8% | |
| [2024-03-26 12:49:07,790] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw | |
| [2024-03-26 12:49:07,790] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using client callable to create LR scheduler | |
| [2024-03-26 12:49:07,790] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = <torch.optim.lr_scheduler.LambdaLR object at 0x7fdb7ac54160> | |
| [2024-03-26 12:49:07,790] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0], mom=[[0.9, 0.999]] | |
| [2024-03-26 12:49:07,791] [INFO] [config.py:984:print] DeepSpeedEngine configuration: | |
| [2024-03-26 12:49:07,791] [INFO] [config.py:988:print] activation_checkpointing_config { | |
| "partition_activations": false, | |
| "contiguous_memory_optimization": false, | |
| "cpu_checkpointing": false, | |
| "number_checkpoints": null, | |
| "synchronize_checkpoint_boundary": false, | |
| "profile": false | |
| } | |
| [2024-03-26 12:49:07,791] [INFO] [config.py:988:print] aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True} | |
| [2024-03-26 12:49:07,791] [INFO] [config.py:988:print] amp_enabled .................. False | |
| [2024-03-26 12:49:07,791] [INFO] [config.py:988:print] amp_params ................... False | |
| [2024-03-26 12:49:07,791] [INFO] [config.py:988:print] autotuning_config ............ { | |
| "enabled": false, | |
| "start_step": null, | |
| "end_step": null, | |
| "metric_path": null, | |
| "arg_mappings": null, | |
| "metric": "throughput", | |
| "model_info": null, | |
| "results_dir": "autotuning_results", | |
| "exps_dir": "autotuning_exps", | |
| "overwrite": true, | |
| "fast": true, | |
| "start_profile_step": 3, | |
| "end_profile_step": 5, | |
| "tuner_type": "gridsearch", | |
| "tuner_early_stopping": 5, | |
| "tuner_num_trials": 50, | |
| "model_info_path": null, | |
| "mp_size": 1, | |
| "max_train_batch_size": null, | |
| "min_train_batch_size": 1, | |
| "max_train_micro_batch_size_per_gpu": 1.024000e+03, | |
| "min_train_micro_batch_size_per_gpu": 1, | |
| "num_tuning_micro_batch_sizes": 3 | |
| } | |
| [2024-03-26 12:49:07,791] [INFO] [config.py:988:print] bfloat16_enabled ............. True | |
| [2024-03-26 12:49:07,791] [INFO] [config.py:988:print] checkpoint_parallel_write_pipeline False | |
| [2024-03-26 12:49:07,791] [INFO] [config.py:988:print] checkpoint_tag_validation_enabled True | |
| [2024-03-26 12:49:07,791] [INFO] [config.py:988:print] checkpoint_tag_validation_fail False | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7fdb73b973d0> | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] communication_data_type ...... None | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}} | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] curriculum_enabled_legacy .... False | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] curriculum_params_legacy ..... False | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}} | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] data_efficiency_enabled ...... False | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] dataloader_drop_last ......... False | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] disable_allgather ............ False | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] dump_state ................... False | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] dynamic_loss_scale_args ...... None | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] eigenvalue_enabled ........... False | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] eigenvalue_gas_boundary_resolution 1 | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] eigenvalue_layer_name ........ bert.encoder.layer | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] eigenvalue_layer_num ......... 0 | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] eigenvalue_max_iter .......... 100 | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] eigenvalue_stability ......... 1e-06 | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] eigenvalue_tol ............... 0.01 | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] eigenvalue_verbose ........... False | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] elasticity_enabled ........... False | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] flops_profiler_config ........ { | |
| "enabled": false, | |
| "recompute_fwd_factor": 0.0, | |
| "profile_step": 1, | |
| "module_depth": -1, | |
| "top_modules": 1, | |
| "detailed": true, | |
| "output_file": null | |
| } | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] fp16_auto_cast ............... None | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] fp16_enabled ................. False | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] fp16_master_weights_and_gradients False | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] global_rank .................. 0 | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] grad_accum_dtype ............. None | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] gradient_accumulation_steps .. 4 | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] gradient_clipping ............ 1.0 | |
| [2024-03-26 12:49:07,792] [INFO] [config.py:988:print] gradient_predivide_factor .... 1.0 | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] graph_harvesting ............. False | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8 | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] initial_dynamic_scale ........ 1 | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] load_universal_checkpoint .... False | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] loss_scale ................... 1.0 | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] memory_breakdown ............. False | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] mics_hierarchial_params_gather False | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] mics_shard_size .............. -1 | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] nebula_config ................ { | |
| "enabled": false, | |
| "persistent_storage_path": null, | |
| "persistent_time_interval": 100, | |
| "num_of_version_in_retention": 2, | |
| "enable_nebula_load": true, | |
| "load_path": null | |
| } | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] optimizer_legacy_fusion ...... False | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] optimizer_name ............... adamw | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] optimizer_params ............. {'lr': 2e-05, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0} | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True} | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] pld_enabled .................. False | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] pld_params ................... False | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] prescale_gradients ........... False | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] scheduler_name ............... None | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] scheduler_params ............. None | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] seq_parallel_communication_data_type torch.float32 | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] sparse_attention ............. None | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] sparse_gradients_enabled ..... False | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] steps_per_print .............. inf | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] train_batch_size ............. 512 | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] train_micro_batch_size_per_gpu 32 | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] use_data_before_expert_parallel_ False | |
| [2024-03-26 12:49:07,793] [INFO] [config.py:988:print] use_node_local_storage ....... False | |
| [2024-03-26 12:49:07,794] [INFO] [config.py:988:print] wall_clock_breakdown ......... False | |
| [2024-03-26 12:49:07,794] [INFO] [config.py:988:print] weight_quantization_config ... None | |
| [2024-03-26 12:49:07,794] [INFO] [config.py:988:print] world_size ................... 4 | |
| [2024-03-26 12:49:07,794] [INFO] [config.py:988:print] zero_allow_untested_optimizer False | |
| [2024-03-26 12:49:07,794] [INFO] [config.py:988:print] zero_config .................. stage=2 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=200000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=200000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True | |
| [2024-03-26 12:49:07,794] [INFO] [config.py:988:print] zero_enabled ................. True | |
| [2024-03-26 12:49:07,794] [INFO] [config.py:988:print] zero_force_ds_cpu_optimizer .. True | |
| [2024-03-26 12:49:07,794] [INFO] [config.py:988:print] zero_optimization_stage ...... 2 | |
| [2024-03-26 12:49:07,794] [INFO] [config.py:974:print_user_config] json = { | |
| "fp16": { | |
| "enabled": false, | |
| "loss_scale": 0, | |
| "loss_scale_window": 1000, | |
| "initial_scale_power": 16, | |
| "hysteresis": 2, | |
| "min_loss_scale": 1 | |
| }, | |
| "bf16": { | |
| "enabled": true | |
| }, | |
| "optimizer": { | |
| "type": "AdamW", | |
| "params": { | |
| "lr": 2e-05, | |
| "betas": [0.9, 0.999], | |
| "eps": 1e-08, | |
| "weight_decay": 0.0 | |
| } | |
| }, | |
| "zero_optimization": { | |
| "stage": 2, | |
| "allgather_partitions": true, | |
| "allgather_bucket_size": 2.000000e+08, | |
| "overlap_comm": true, | |
| "reduce_scatter": true, | |
| "reduce_bucket_size": 2.000000e+08, | |
| "contiguous_gradients": true | |
| }, | |
| "gradient_accumulation_steps": 4, | |
| "gradient_clipping": 1.0, | |
| "steps_per_print": inf, | |
| "train_batch_size": 512, | |
| "train_micro_batch_size_per_gpu": 32, | |
| "wall_clock_breakdown": false | |
| } | |
| [INFO|trainer.py:1812] 2024-03-26 12:49:07,794 >> ***** Running training ***** | |
| [INFO|trainer.py:1813] 2024-03-26 12:49:07,794 >> Num examples = 9,229 | |
| [INFO|trainer.py:1814] 2024-03-26 12:49:07,794 >> Num Epochs = 3 | |
| [INFO|trainer.py:1815] 2024-03-26 12:49:07,794 >> Instantaneous batch size per device = 32 | |
| [INFO|trainer.py:1818] 2024-03-26 12:49:07,794 >> Total train batch size (w. parallel, distributed & accumulation) = 512 | |
| [INFO|trainer.py:1819] 2024-03-26 12:49:07,794 >> Gradient Accumulation steps = 4 | |
| [INFO|trainer.py:1820] 2024-03-26 12:49:07,794 >> Total optimization steps = 54 | |
| [INFO|trainer.py:1821] 2024-03-26 12:49:07,795 >> Number of trainable parameters = 6,738,415,616 | |
| 0%| | 0/54 [00:00<?, ?it/s] 2%|▏ | 1/54 [00:44<39:00, 44.17s/it] 4%|▎ | 2/54 [01:22<35:33, 41.03s/it] 6%|▌ | 3/54 [02:02<34:06, 40.14s/it] 7%|▋ | 4/54 [02:42<33:41, 40.44s/it] 9%|▉ | 5/54 [03:30<35:09, 43.06s/it] 11%|█ | 6/54 [04:06<32:26, 40.55s/it] 13%|█▎ | 7/54 [04:51<33:02, 42.19s/it] 15%|█▍ | 8/54 [05:34<32:30, 42.39s/it] 17%|█▋ | 9/54 [06:08<29:44, 39.66s/it] 19%|█▊ | 10/54 [06:46<28:46, 39.23s/it] {'loss': 1.0908, 'grad_norm': 1.638972731060276, 'learning_rate': 1.9659258262890683e-05, 'epoch': 0.55} | |
| 19%|█▊ | 10/54 [06:46<28:46, 39.23s/it] 20%|██ | 11/54 [07:18<26:32, 37.04s/it] 22%|██▏ | 12/54 [08:07<28:23, 40.57s/it] 24%|██▍ | 13/54 [08:47<27:43, 40.58s/it] 26%|██▌ | 14/54 [09:27<26:54, 40.35s/it] 28%|██▊ | 15/54 [10:10<26:41, 41.05s/it] 30%|██▉ | 16/54 [10:54<26:30, 41.85s/it] 31%|███▏ | 17/54 [11:30<24:51, 40.30s/it] 33%|███▎ | 18/54 [12:17<25:17, 42.15s/it] 35%|███▌ | 19/54 [13:04<25:26, 43.60s/it] 37%|███▋ | 20/54 [13:41<23:41, 41.81s/it] {'loss': 0.9545, 'grad_norm': 1.0710882452469266, 'learning_rate': 1.608761429008721e-05, 'epoch': 1.1} | |
| 37%|███▋ | 20/54 [13:41<23:41, 41.81s/it] 39%|███▉ | 21/54 [14:18<22:11, 40.34s/it] 41%|████ | 22/54 [14:58<21:27, 40.23s/it] 43%|████▎ | 23/54 [15:43<21:26, 41.50s/it] 44%|████▍ | 24/54 [16:20<20:05, 40.17s/it] 46%|████▋ | 25/54 [17:02<19:40, 40.71s/it] 48%|████▊ | 26/54 [17:42<18:58, 40.66s/it] 50%|█████ | 27/54 [18:25<18:35, 41.32s/it] 52%|█████▏ | 28/54 [19:06<17:46, 41.03s/it] 54%|█████▎ | 29/54 [19:46<17:00, 40.83s/it] 56%|█████▌ | 30/54 [20:28<16:30, 41.27s/it] {'loss': 0.8724, 'grad_norm': 0.9859220972082218, 'learning_rate': 1e-05, 'epoch': 1.64} | |
| 56%|█████▌ | 30/54 [20:28<16:30, 41.27s/it] 57%|█████▋ | 31/54 [21:07<15:28, 40.38s/it] 59%|█████▉ | 32/54 [21:47<14:51, 40.52s/it] 61%|██████ | 33/54 [22:34<14:49, 42.38s/it] 63%|██████▎ | 34/54 [23:11<13:35, 40.79s/it] 65%|██████▍ | 35/54 [23:57<13:23, 42.28s/it] 67%|██████▋ | 36/54 [24:35<12:20, 41.14s/it] 69%|██████▊ | 37/54 [25:09<11:02, 38.94s/it] 70%|███████ | 38/54 [25:49<10:26, 39.14s/it] 72%|███████▏ | 39/54 [26:28<09:46, 39.10s/it] 74%|███████▍ | 40/54 [27:07<09:07, 39.10s/it] {'loss': 0.8427, 'grad_norm': 1.1753998705017632, 'learning_rate': 3.912385709912794e-06, 'epoch': 2.19} | |
| 74%|███████▍ | 40/54 [27:07<09:07, 39.10s/it] 76%|███████▌ | 41/54 [27:58<09:14, 42.62s/it] 78%|███████▊ | 42/54 [28:34<08:07, 40.63s/it] 80%|███████▉ | 43/54 [29:19<07:41, 41.94s/it] 81%|████████▏ | 44/54 [30:01<07:01, 42.15s/it] 83%|████████▎ | 45/54 [30:36<05:59, 39.99s/it] 85%|████████▌ | 46/54 [31:19<05:26, 40.83s/it] 87%|████████▋ | 47/54 [32:06<04:57, 42.54s/it] 89%|████████▉ | 48/54 [32:56<04:29, 44.97s/it] 91%|█████████ | 49/54 [33:37<03:38, 43.65s/it] 93%|█████████▎| 50/54 [34:11<02:43, 40.79s/it] {'loss': 0.8133, 'grad_norm': 1.1336143023825, 'learning_rate': 3.4074173710931804e-07, 'epoch': 2.74} | |
| 93%|█████████▎| 50/54 [34:11<02:43, 40.79s/it] 94%|█████████▍| 51/54 [34:54<02:04, 41.36s/it] 96%|█████████▋| 52/54 [35:31<01:20, 40.18s/it] 98%|█████████▊| 53/54 [36:08<00:39, 39.33s/it] 100%|██████████| 54/54 [36:47<00:00, 39.03s/it][INFO|trainer.py:2067] 2024-03-26 13:25:55,119 >> | |
| Training completed. Do not forget to share your model on huggingface.co/models =) | |
| {'train_runtime': 2207.3234, 'train_samples_per_second': 12.543, 'train_steps_per_second': 0.024, 'train_loss': 0.9069184638835766, 'epoch': 2.96} | |
| 100%|██████████| 54/54 [36:47<00:00, 39.03s/it] 100%|██████████| 54/54 [36:47<00:00, 40.88s/it] | |
| [INFO|trainer.py:3067] 2024-03-26 13:26:02,466 >> Saving model checkpoint to /home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909 | |
| [INFO|configuration_utils.py:473] 2024-03-26 13:26:02,467 >> Configuration saved in /home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909/config.json | |
| [INFO|configuration_utils.py:614] 2024-03-26 13:26:02,468 >> Configuration saved in /home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909/generation_config.json | |
| [2024-03-26 13:26:04,855] [INFO] [launch.py:347:main] Process 163483 exits successfully. | |
| [2024-03-26 13:26:04,855] [INFO] [launch.py:347:main] Process 163485 exits successfully. | |
| [2024-03-26 13:26:04,855] [INFO] [launch.py:347:main] Process 163484 exits successfully. | |
| [INFO|modeling_utils.py:2462] 2024-03-26 13:26:17,853 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 3 checkpoint shards. You can find where each parameters has been saved in the index located at /home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909/model.safetensors.index.json. | |
| [INFO|tokenization_utils_base.py:2459] 2024-03-26 13:26:17,854 >> tokenizer config file saved in /home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909/tokenizer_config.json | |
| [INFO|tokenization_utils_base.py:2468] 2024-03-26 13:26:17,854 >> Special tokens file saved in /home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909/special_tokens_map.json | |
| ***** train metrics ***** | |
| epoch = 2.96 | |
| train_loss = 0.9069 | |
| train_runtime = 0:36:47.32 | |
| train_samples_per_second = 12.543 | |
| train_steps_per_second = 0.024 | |
| Figure saved: /home/lirenhao/projects/LLaMA-Factory/output/d2417a68-3122-41af-9d26-a7f736043909/training_loss.png | |
| 03/26/2024 13:26:18 - WARNING - llmtuner.extras.ploting - No metric eval_loss to plot. | |
| [INFO|modelcard.py:450] 2024-03-26 13:26:18,360 >> Dropping the following result as it does not have all the necessary fields: | |
| {'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}} | |
| [2024-03-26 13:26:20,872] [INFO] [launch.py:347:main] Process 163482 exits successfully. | |