Model parameters: d_model 768 ffw_size 3072 kv_size 64 n_heads 12 n_layers 15 Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 15 --hidden-size 768 --num-attention-heads 12 --kv-channels 64 --ffn-hidden-size 3072 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 4 --global-batch-size 256 --train-samples 44_416_143 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-146m91b100mdedup --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 44_416_143 --lr-warmup-samples 444_161 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 100 --save-interval 10000 --eval-interval 10000 --eval-iters 1 --tensorboard-dir tensorboard_146m91b100mdedup --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_146m91b100mdedup --load checkpoints_146m91b100mdedup --train-weighted-split-paths-path train100mdedup.txt --valid-weighted-split-paths-path val.txt --data-impl mmap --deepspeed --deepspeed_config ds_configs/3328804.json --zero-stage 0 START 3328804: Fri 17 Mar 2023 10:52:16 AM EET 0: 0: 0: ======================= ROCm System Management Interface ======================= 0: ================================= Concise Info ================================= 0: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 0: 0 41.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 0: 1 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 0: 2 40.0c 85.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 0: 3 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 0: 4 42.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 0: 5 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 0: 6 39.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 0: 7 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 0: ================================================================================ 0: ============================= End of ROCm SMI Log ============================== 6: 6: 6: ======================= ROCm System Management Interface ======================= 6: ================================= Concise Info ================================= 6: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 6: 0 45.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 6: 1 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 6: 2 47.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 6: 3 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 6: 4 44.0c 97.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 6: 5 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 6: 6 39.0c 94.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 6: 7 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 6: ================================================================================ 6: ============================= End of ROCm SMI Log ============================== 4: 4: 4: ======================= ROCm System Management Interface ======================= 4: ================================= Concise Info ================================= 4: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 4: 0 41.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 4: 1 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 4: 2 45.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 4: 3 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 4: 4 40.0c 96.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 4: 5 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 4: 6 39.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 4: 7 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 4: ================================================================================ 4: ============================= End of ROCm SMI Log ============================== 1: 1: 1: ======================= ROCm System Management Interface ======================= 1: ================================= Concise Info ================================= 1: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 1: 0 45.0c 95.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 1: 1 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 1: 2 39.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 1: 3 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 1: 4 47.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 1: 5 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 1: 6 42.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 1: 7 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 1: ================================================================================ 1: ============================= End of ROCm SMI Log ============================== 7: 7: 7: ======================= ROCm System Management Interface ======================= 7: ================================= Concise Info ================================= 7: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 7: 0 46.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 7: 1 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 7: 2 46.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 7: 3 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 7: 4 47.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 7: 5 40.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 7: 6 43.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 7: 7 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 7: ================================================================================ 7: ============================= End of ROCm SMI Log ============================== 3: 3: 3: ======================= ROCm System Management Interface ======================= 3: ================================= Concise Info ================================= 3: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 3: 0 47.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 3: 1 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 3: 2 41.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 3: 3 41.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 3: 4 46.0c 83.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 3: 5 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 3: 6 40.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 3: 7 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 3: ================================================================================ 3: ============================= End of ROCm SMI Log ============================== 2: 2: 2: ======================= ROCm System Management Interface ======================= 2: ================================= Concise Info ================================= 2: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2: 0 45.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2: 1 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2: 2 45.0c 99.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2: 3 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2: 4 43.0c 97.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2: 5 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2: 6 43.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2: 7 38.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2: ================================================================================ 2: ============================= End of ROCm SMI Log ============================== 5: 5: 5: ======================= ROCm System Management Interface ======================= 5: ================================= Concise Info ================================= 5: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 5: 0 42.0c 94.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 5: 1 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 5: 2 43.0c 90.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 5: 3 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 5: 4 46.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 5: 5 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 5: 6 41.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 5: 7 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 5: ================================================================================ 5: ============================= End of ROCm SMI Log ============================== 0: Launching on nid005684 (0/8), master nid005684 port 9999, GPUs 8, CUDA: True 6: Launching on nid005690 (6/8), master nid005684 port 9999, GPUs 8, CUDA: True 3: Launching on nid005687 (3/8), master nid005684 port 9999, GPUs 8, CUDA: True 1: Launching on nid005685 (1/8), master nid005684 port 9999, GPUs 8, CUDA: True 2: Launching on nid005686 (2/8), master nid005684 port 9999, GPUs 8, CUDA: True 4: Launching on nid005688 (4/8), master nid005684 port 9999, GPUs 8, CUDA: True 5: Launching on nid005689 (5/8), master nid005684 port 9999, GPUs 8, CUDA: True 7: Launching on nid005691 (7/8), master nid005684 port 9999, GPUs 8, CUDA: True 0: using world size: 64, data-parallel-size: 64, tensor-model-parallel size: 1, pipeline-model-parallel size: 1 0: accumulate and all-reduce gradients in fp32 for bfloat16 data type. 0: using torch.bfloat16 for parameters ... 0: ------------------------ arguments ------------------------ 0: abort_on_unmet_fused_kernel_constraints ......... False 0: accumulate_allreduce_grads_in_fp32 .............. True 0: adam_beta1 ...................................... 0.9 0: adam_beta2 ...................................... 0.999 0: adam_eps ........................................ 1e-08 0: adlr_autoresume ................................. False 0: adlr_autoresume_interval ........................ 1000 0: apply_query_key_layer_scaling ................... True 0: apply_residual_connection_post_layernorm ........ False 0: attention_dropout ............................... 0.1 0: attention_softmax_in_fp32 ....................... False 0: bert_binary_head ................................ True 0: bert_load ....................................... None 0: bf16 ............................................ True 0: bias_dropout_fusion ............................. True 0: bias_gelu_fusion ................................ True 0: biencoder_projection_dim ........................ 0 0: biencoder_shared_query_context_model ............ False 0: block_data_path ................................. None 0: checkpoint_activations .......................... True 0: checkpoint_in_cpu ............................... False 0: checkpoint_num_layers ........................... 1 0: clip_grad ....................................... 1.0 0: codecarbon_dir .................................. None 0: consumed_train_samples .......................... 0 0: consumed_train_tokens ........................... 0 0: consumed_valid_samples .......................... 0 0: contigious_checkpointing ........................ False 0: cpu_optimizer ................................... False 0: cpu_torch_adam .................................. False 0: curriculum_learning ............................. False 0: data_impl ....................................... mmap 0: data_parallel_size .............................. 64 0: data_path ....................................... None 0: dataloader_type ................................. single 0: DDP_impl ........................................ local 0: decoder_seq_length .............................. None 0: deepscale ....................................... False 0: deepscale_config ................................ None 0: deepspeed ....................................... True 0: deepspeed_activation_checkpointing .............. False 0: deepspeed_config ................................ ds_configs/3328804.json 0: deepspeed_mpi ................................... False 0: distribute_checkpointed_activations ............. False 0: distributed_backend ............................. nccl 0: embed_layernorm ................................. False 0: embedding_path .................................. None 0: encoder_seq_length .............................. 2048 0: eod_mask_loss ................................... False 0: eval_interval ................................... 10000 0: eval_iters ...................................... 1 0: eval_only ....................................... None 0: evidence_data_path .............................. None 0: exit_duration_in_mins ........................... None 0: exit_interval ................................... None 0: ffn_hidden_size ................................. 3072 0: finetune ........................................ False 0: fp16 ............................................ False 0: fp16_lm_cross_entropy ........................... False 0: fp32_residual_connection ........................ False 0: gigaflos_no_embeds .............................. 0 0: global_batch_size ............................... 256 0: glu_activation .................................. None 0: hidden_dropout .................................. 0.1 0: hidden_size ..................................... 768 0: hysteresis ...................................... 2 0: ict_head_size ................................... None 0: ict_load ........................................ None 0: img_dim ......................................... 224 0: indexer_batch_size .............................. 128 0: indexer_log_interval ............................ 1000 0: inference ....................................... False 0: init_method_std ................................. 0.02 0: init_method_xavier_uniform ...................... False 0: initial_loss_scale .............................. 4294967296 0: kill_switch_path ................................ kill-switch-146m91b100mdedup 0: kv_channels ..................................... 64 0: layer_norm_fusion ............................... True 0: layernorm_epsilon ............................... 1e-05 0: lazy_mpu_init ................................... None 0: load ............................................ checkpoints_146m91b100mdedup 0: local_rank ...................................... None 0: log_batch_size_to_tensorboard ................... True 0: log_interval .................................... 100 0: log_learning_rate_to_tensorboard ................ True 0: log_level ....................................... None 0: log_level_replica ............................... None 0: log_loss_scale_to_tensorboard ................... True 0: log_num_zeros_in_grad ........................... False 0: log_params_norm ................................. False 0: log_path ........................................ None 0: log_timers_to_tensorboard ....................... True 0: log_validation_ppl_to_tensorboard ............... True 0: loss_on_targets_only ............................ False 0: loss_scale ...................................... 12.0 0: loss_scale_window ............................... 1000 0: lr .............................................. 0.0002 0: lr_decay_iters .................................. None 0: lr_decay_samples ................................ 44416143 0: lr_decay_style .................................. cosine 0: lr_decay_tokens ................................. None 0: lr_warmup_fraction .............................. None 0: lr_warmup_iters ................................. 0 0: lr_warmup_samples ............................... 444161 0: make_vocab_size_divisible_by .................... 128 0: mask_prob ....................................... 0.15 0: masked_softmax_fusion ........................... True 0: max_position_embeddings ......................... 2048 0: mean_noise_span_length .......................... None 0: memory_centric_tiled_linear ..................... False 0: merge_file ...................................... gpt2/merges.txt 0: micro_batch_size ................................ 4 0: min_loss_scale .................................. 1.0 0: min_lr .......................................... 2e-05 0: mmap_warmup ..................................... False 0: no_load_optim ................................... None 0: no_load_rng ..................................... None 0: no_save_optim ................................... None 0: no_save_rng ..................................... None 0: noise_density ................................... None 0: num_attention_heads ............................. 12 0: num_channels .................................... 3 0: num_classes ..................................... 1000 0: num_layers ...................................... 15 0: num_layers_per_virtual_pipeline_stage ........... None 0: num_workers ..................................... 2 0: onnx_safe ....................................... None 0: openai_gelu ..................................... False 0: optimizer ....................................... adam 0: optimizer_fusion ................................ True 0: override_lr_scheduler ........................... False 0: pad_vocab_size_to ............................... None 0: params_dtype .................................... torch.bfloat16 0: partition_activations ........................... False 0: patch_dim ....................................... 16 0: pipeline_model_parallel_size .................... 1 0: position_embedding_type ......................... PositionEmbeddingType.absolute 0: pp_partition_method ............................. None 0: profile_backward ................................ False 0: query_in_block_prob ............................. 0.1 0: rampup_batch_size ............................... None 0: rank ............................................ 0 0: remote_device ................................... none 0: reset_attention_mask ............................ False 0: reset_position_ids .............................. False 0: reset_progress .................................. None 0: retriever_report_topk_accuracies ................ [] 0: retriever_score_scaling ......................... False 0: retriever_seq_length ............................ 256 0: reweight_loss_based_on_position_frequency ....... False 0: sample_rate ..................................... 1.0 0: save ............................................ checkpoints_146m91b100mdedup 0: save_interval ................................... 10000 0: scatter_gather_tensors_in_pipeline .............. True 0: scattered_embeddings ............................ False 0: seed ............................................ 1234 0: seq_length ...................................... 2048 0: sgd_momentum .................................... 0.9 0: short_seq_prob .................................. 0.1 0: skip_train_iteration_range ...................... None 0: split ........................................... None 0: split_transformers .............................. False 0: sync_tp_duplicated_parameters ................... False 0: synchronize_each_layer .......................... False 0: tensor_model_parallel_size ...................... 1 0: tensorboard_dir ................................. tensorboard_146m91b100mdedup 0: tensorboard_log_interval ........................ 1 0: tensorboard_queue_size .......................... 5 0: test_weighted_split_paths ....................... None 0: test_weighted_split_paths_path .................. None 0: tile_factor ..................................... 1 0: titles_data_path ................................ None 0: tokenizer_name_or_path .......................... None 0: tokenizer_type .................................. GPT2BPETokenizer 0: train_iters ..................................... None 0: train_samples ................................... 44416143 0: train_tokens .................................... None 0: train_weighted_split_names ...................... ['train'] 0: train_weighted_split_paths ...................... [['/scratch/project_462000119/data/c4_subsampled/gpt2tok_c4_en_dedup_100M_text_document']] 0: train_weighted_split_paths_path ................. None 0: train_weighted_split_splits ..................... [['0:1']] 0: train_weighted_split_weights .................... [['1.0']] 0: universal_checkpoint ............................ False 0: use_bnb_optimizer ............................... False 0: use_checkpoint_lr_scheduler ..................... False 0: use_contiguous_buffers_in_ddp ................... True 0: use_cpu_initialization .......................... None 0: use_one_sent_docs ............................... False 0: use_pin_memory .................................. False 0: valid_num_workers ............................... 2 0: valid_weighted_split_names ...................... ['validation'] 0: valid_weighted_split_paths ...................... [['/scratch/project_462000119/data/c4_validation/gpt2tok_c4validation_rerun_text_document']] 0: valid_weighted_split_paths_path ................. None 0: valid_weighted_split_splits ..................... [['0:1']] 0: valid_weighted_split_weights .................... [['1.0']] 0: virtual_pipeline_model_parallel_size ............ None 0: vocab_extra_ids ................................. 0 0: vocab_file ...................................... gpt2/vocab.json 0: weight_decay .................................... 0.1 0: world_size ...................................... 64 0: zero_allgather_bucket_size ...................... 0.0 0: zero_contigious_gradients ....................... False 0: zero_reduce_bucket_size ......................... 0.0 0: zero_reduce_scatter ............................. False 0: zero_stage ...................................... 0 0: -------------------- end of arguments --------------------- 0: setting number of micro-batches to constant 1 0: > building GPT2BPETokenizer tokenizer ... 0: > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304) 0: DeepSpeed general environment info: 0: torch install path ............... ['/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch'] 0: torch version .................... 1.13.0+rocm5.2 0: torch cuda version ............... None 0: torch hip version ................ 5.2.21151-afdc89f8 0: nvcc version ..................... None 0: deepspeed install path ........... ['/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/deepspeed'] 0: deepspeed info ................... 0.7.5, unknown, unknown 0: deepspeed wheel compiled w. ...... torch 1.13, hip 5.1 0: **** Git info for Megatron: git_hash=unknown git_branch=unknown **** 0: > initializing torch distributed ... 0: [2023-03-17 10:54:15,674] [INFO] [comm.py:633:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl 7: > setting tensorboard ... 0: > initializing tensor model parallel with size 1 0: > initializing pipeline model parallel with size 1 0: > setting random seeds to 1234 ... 0: > initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 3952 and data parallel seed: 1234 0: > compiling dataset index builder ... 0: make: Entering directory '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/data' 0: make: Nothing to be done for 'default'. 0: make: Leaving directory '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/data' 0: >>> done with dataset index builder. Compilation time: 0.103 seconds 0: > compiling and loading fused kernels ... 0: >>> done with compiling and loading fused kernels. Compilation time: 32.292 seconds 0: time to initialize megatron (seconds): 90.888 0: [after megatron is initialized] datetime: 2023-03-17 10:54:50 0: building GPT model ... 0: [2023-03-17 10:54:51,021] [INFO] [utils.py:827:see_memory_usage] Before Building Model 0: [2023-03-17 10:54:51,022] [INFO] [utils.py:828:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.0 GB Max_CA 0 GB 0: [2023-03-17 10:54:51,022] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 30.62 GB, percent = 6.1% 0: SEED_LAYERS=False BASE_SEED=1234 SEED_FN=None 0: Using topology: {ProcessCoord(pipe=0, data=0, model=0): 0, ProcessCoord(pipe=0, data=1, model=0): 1, ProcessCoord(pipe=0, data=2, model=0): 2, ProcessCoord(pipe=0, data=3, model=0): 3, ProcessCoord(pipe=0, data=4, model=0): 4, ProcessCoord(pipe=0, data=5, model=0): 5, ProcessCoord(pipe=0, data=6, model=0): 6, ProcessCoord(pipe=0, data=7, model=0): 7, ProcessCoord(pipe=0, data=8, model=0): 8, ProcessCoord(pipe=0, data=9, model=0): 9, ProcessCoord(pipe=0, data=10, model=0): 10, ProcessCoord(pipe=0, data=11, model=0): 11, ProcessCoord(pipe=0, data=12, model=0): 12, ProcessCoord(pipe=0, data=13, model=0): 13, ProcessCoord(pipe=0, data=14, model=0): 14, ProcessCoord(pipe=0, data=15, model=0): 15, ProcessCoord(pipe=0, data=16, model=0): 16, ProcessCoord(pipe=0, data=17, model=0): 17, ProcessCoord(pipe=0, data=18, model=0): 18, ProcessCoord(pipe=0, data=19, model=0): 19, ProcessCoord(pipe=0, data=20, model=0): 20, ProcessCoord(pipe=0, data=21, model=0): 21, ProcessCoord(pipe=0, data=22, model=0): 22, ProcessCoord(pi 0: pe=0, data=23, model=0): 23, ProcessCoord(pipe=0, data=24, model=0): 24, ProcessCoord(pipe=0, data=25, model=0): 25, ProcessCoord(pipe=0, data=26, model=0): 26, ProcessCoord(pipe=0, data=27, model=0): 27, ProcessCoord(pipe=0, data=28, model=0): 28, ProcessCoord(pipe=0, data=29, model=0): 29, ProcessCoord(pipe=0, data=30, model=0): 30, ProcessCoord(pipe=0, data=31, model=0): 31, ProcessCoord(pipe=0, data=32, model=0): 32, ProcessCoord(pipe=0, data=33, model=0): 33, ProcessCoord(pipe=0, data=34, model=0): 34, ProcessCoord(pipe=0, data=35, model=0): 35, ProcessCoord(pipe=0, data=36, model=0): 36, ProcessCoord(pipe=0, data=37, model=0): 37, ProcessCoord(pipe=0, data=38, model=0): 38, ProcessCoord(pipe=0, data=39, model=0): 39, ProcessCoord(pipe=0, data=40, model=0): 40, ProcessCoord(pipe=0, data=41, model=0): 41, ProcessCoord(pipe=0, data=42, model=0): 42, ProcessCoord(pipe=0, data=43, model=0): 43, ProcessCoord(pipe=0, data=44, model=0): 44, ProcessCoord(pipe=0, data=45, model=0): 45, ProcessCoord(pipe=0, data=4 0: 6, model=0): 46, ProcessCoord(pipe=0, data=47, model=0): 47, ProcessCoord(pipe=0, data=48, model=0): 48, ProcessCoord(pipe=0, data=49, model=0): 49, ProcessCoord(pipe=0, data=50, model=0): 50, ProcessCoord(pipe=0, data=51, model=0): 51, ProcessCoord(pipe=0, data=52, model=0): 52, ProcessCoord(pipe=0, data=53, model=0): 53, ProcessCoord(pipe=0, data=54, model=0): 54, ProcessCoord(pipe=0, data=55, model=0): 55, ProcessCoord(pipe=0, data=56, model=0): 56, ProcessCoord(pipe=0, data=57, model=0): 57, ProcessCoord(pipe=0, data=58, model=0): 58, ProcessCoord(pipe=0, data=59, model=0): 59, ProcessCoord(pipe=0, data=60, model=0): 60, ProcessCoord(pipe=0, data=61, model=0): 61, ProcessCoord(pipe=0, data=62, model=0): 62, ProcessCoord(pipe=0, data=63, model=0): 63} 0: [2023-03-17 10:54:53,043] [INFO] [module.py:366:_partition_layers] Partitioning pipeline stages with method type:transformer 0: stage=0 layers=22 0: 0: _to_float16 0: 1: EmbeddingPipe 0: 2: 0: 3: ParallelTransformerLayerPipe 0: 4: ParallelTransformerLayerPipe 0: 5: ParallelTransformerLayerPipe 0: 6: ParallelTransformerLayerPipe 0: 7: ParallelTransformerLayerPipe 0: 8: ParallelTransformerLayerPipe 0: 9: ParallelTransformerLayerPipe 0: 10: ParallelTransformerLayerPipe 0: 11: ParallelTransformerLayerPipe 0: 12: ParallelTransformerLayerPipe 0: 13: ParallelTransformerLayerPipe 0: 14: ParallelTransformerLayerPipe 0: 15: ParallelTransformerLayerPipe 0: 16: ParallelTransformerLayerPipe 0: 17: ParallelTransformerLayerPipe 0: 18: undo 0: 19: MixedFusedLayerNorm 0: 20: EmbeddingPipe 0: 21: float16_to_fp32 0: loss: CrossEntropy 0: [2023-03-17 10:54:53,259] [INFO] [utils.py:827:see_memory_usage] After Building Model 0: [2023-03-17 10:54:53,260] [INFO] [utils.py:828:see_memory_usage] MA 0.28 GB Max_MA 0.28 GB CA 0.29 GB Max_CA 0 GB 0: [2023-03-17 10:54:53,260] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 30.64 GB, percent = 6.1% 0: setting training iterations to 173500 0: > learning rate decay style: cosine 0: DeepSpeed is enabled. 0: [2023-03-17 10:54:53,262] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed info: version=0.7.5, git-hash=unknown, git-branch=unknown 0: [2023-03-17 10:55:05,928] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False 0: [2023-03-17 10:55:05,928] [INFO] [logging.py:68:log_dist] [Rank 0] Removing param_group that has no 'params' in the client Optimizer 0: [2023-03-17 10:55:05,929] [INFO] [logging.py:68:log_dist] [Rank 0] Using client Optimizer as basic optimizer 0: [2023-03-17 10:55:05,933] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam 0: [2023-03-17 10:55:05,933] [INFO] [logging.py:68:log_dist] [Rank 0] Creating BF16 optimizer 0: [2023-03-17 10:55:06,058] [INFO] [utils.py:827:see_memory_usage] begin bf16_optimizer 0: [2023-03-17 10:55:06,058] [INFO] [utils.py:828:see_memory_usage] MA 0.28 GB Max_MA 0.29 GB CA 0.31 GB Max_CA 0 GB 0: [2023-03-17 10:55:06,058] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.33 GB, percent = 6.2% 0: ninja: no work to do. 0: Time to load utils op: 0.17024493217468262 seconds 0: Time to load utils op: 0.0009210109710693359 seconds 0: Time to load utils op: 0.3025174140930176 seconds 0: Time to load utils op: 0.40522170066833496 seconds 0: Time to load utils op: 0.40535449981689453 seconds 0: Time to load utils op: 0.40515851974487305 seconds 0: Time to load utils op: 0.4055020809173584 seconds 0: Time to load utils op: 0.40581440925598145 seconds 0: Time to load utils op: 0.40638136863708496 seconds 3: Time to load utils op: 0.41220831871032715 secondsTime to load utils op: 0.4131135940551758 seconds 3: 3: Time to load utils op: 0.41216421127319336 seconds 3: Time to load utils op: 0.41222667694091797 seconds 3: Time to load utils op: 0.4121713638305664 secondsTime to load utils op: 0.41235780715942383 seconds 3: 3: Time to load utils op: 0.41218090057373047 secondsTime to load utils op: 0.4130063056945801 seconds 3: 4: Time to load utils op: 0.4123718738555908 seconds 4: Time to load utils op: 0.4123961925506592 seconds 4: Time to load utils op: 0.41291189193725586 seconds 4: Time to load utils op: 0.41266942024230957 seconds 4: Time to load utils op: 0.4129307270050049 secondsTime to load utils op: 0.4126582145690918 seconds 4: 4: Time to load utils op: 0.4122953414916992 seconds 4: Time to load utils op: 0.41281580924987793 seconds 1: Time to load utils op: 0.41287970542907715 secondsTime to load utils op: 0.41288018226623535 seconds 1: 1: Time to load utils op: 0.41290903091430664 seconds 1: Time to load utils op: 0.4129207134246826 secondsTime to load utils op: 0.4129335880279541 secondsTime to load utils op: 0.4129214286804199 seconds 1: 1: 1: Time to load utils op: 0.41294169425964355 secondsTime to load utils op: 0.4129319190979004 seconds 1: 2: Time to load utils op: 0.412341833114624 secondsTime to load utils op: 0.4123504161834717 seconds 2: 2: Time to load utils op: 0.4123518466949463 seconds 2: Time to load utils op: 0.41236019134521484 secondsTime to load utils op: 0.41235804557800293 seconds 2: 2: Time to load utils op: 0.4123697280883789 seconds 2: Time to load utils op: 0.41237854957580566 seconds 2: Time to load utils op: 0.41237807273864746 seconds 5: Time to load utils op: 0.4122605323791504 secondsTime to load utils op: 0.41226649284362793 seconds 5: 5: Time to load utils op: 0.4122655391693115 seconds 5: Time to load utils op: 0.4123051166534424 secondsTime to load utils op: 0.41230297088623047 seconds 5: Time to load utils op: 0.4123098850250244 seconds 5: 5: Time to load utils op: 0.41228508949279785 secondsTime to load utils op: 0.4123117923736572 seconds 5: 0: Time to load utils op: 0.0005087852478027344 seconds 0: Time to load utils op: 0.0005815029144287109 seconds 0: Time to load utils op: 0.0005743503570556641 seconds 6: Time to load utils op: 0.411773681640625 secondsTime to load utils op: 0.4118378162384033 seconds 6: 0: Time to load utils op: 0.0004000663757324219 seconds 6: Time to load utils op: 0.41184449195861816 seconds 6: Time to load utils op: 0.4118466377258301 seconds 6: Time to load utils op: 0.41185426712036133 seconds 6: Time to load utils op: 0.4118678569793701 seconds 6: Time to load utils op: 0.4118802547454834 seconds 0: Time to load utils op: 0.0003910064697265625 seconds 6: Time to load utils op: 0.4118969440460205 seconds 0: Time to load utils op: 0.0003883838653564453 seconds 7: Time to load utils op: 0.411837100982666 seconds 7: Time to load utils op: 0.41184353828430176 seconds 7: Time to load utils op: 0.41185498237609863 seconds 7: Time to load utils op: 0.4118642807006836 secondsTime to load utils op: 0.4118344783782959 seconds 7: Time to load utils op: 0.411853551864624 seconds 7: 7: Time to load utils op: 0.41187405586242676 secondsTime to load utils op: 0.41187262535095215 seconds 7: 3: Time to load utils op: 0.0008447170257568359 seconds 3: Time to load utils op: 0.0008676052093505859 seconds 3: Time to load utils op: 0.0008158683776855469 seconds 3: Time to load utils op: 0.0009341239929199219 seconds 3: Time to load utils op: 0.0008814334869384766 secondsTime to load utils op: 0.0009140968322753906 seconds 3: 3: Time to load utils op: 0.0008928775787353516 seconds 3: Time to load utils op: 0.0010094642639160156 seconds 4: Time to load utils op: 0.0007474422454833984 seconds 4: Time to load utils op: 0.0007131099700927734 seconds 4: Time to load utils op: 0.0007634162902832031 seconds 4: Time to load utils op: 0.0007886886596679688 seconds 4: Time to load utils op: 0.0010335445404052734 secondsTime to load utils op: 0.0010180473327636719 seconds 4: 4: Time to load utils op: 0.0009579658508300781 seconds 4: Time to load utils op: 0.0010213851928710938 seconds 1: Time to load utils op: 0.0009188652038574219 seconds 6: Time to load utils op: 0.0005614757537841797 seconds 6: Time to load utils op: 0.00048422813415527344 secondsTime to load utils op: 0.0004730224609375 seconds 6: 6: Time to load utils op: 0.0004520416259765625 seconds 6: Time to load utils op: 0.00047135353088378906 secondsTime to load utils op: 0.00046563148498535156 secondsTime to load utils op: 0.0004489421844482422 seconds 6: 6: 1: Time to load utils op: 0.0012967586517333984 seconds 1: Time to load utils op: 0.0012900829315185547 secondsTime to load utils op: 0.0012240409851074219 seconds 1: 6: Time to load utils op: 0.0005929470062255859 seconds 1: Time to load utils op: 0.0012738704681396484 secondsTime to load utils op: 0.001344919204711914 seconds 1: 1: Time to load utils op: 0.0013384819030761719 seconds 1: Time to load utils op: 0.0013659000396728516 seconds 2: Time to load utils op: 0.0006933212280273438 seconds 2: Time to load utils op: 0.0009324550628662109 seconds 2: Time to load utils op: 0.0011744499206542969 seconds 2: Time to load utils op: 0.0011713504791259766 seconds 2: Time to load utils op: 0.0011322498321533203 seconds 2: Time to load utils op: 0.0011386871337890625 seconds 2: Time to load utils op: 0.001104116439819336 seconds 2: Time to load utils op: 0.0011837482452392578 seconds 5: Time to load utils op: 0.0008375644683837891 seconds 5: Time to load utils op: 0.0009138584136962891 seconds 5: Time to load utils op: 0.0008924007415771484 seconds 5: Time to load utils op: 0.00099945068359375 seconds 5: Time to load utils op: 0.0010287761688232422 seconds 5: Time to load utils op: 0.0011341571807861328 secondsTime to load utils op: 0.0011751651763916016 seconds 5: 5: Time to load utils op: 0.0011954307556152344 seconds 7: Time to load utils op: 0.0008435249328613281 seconds 7: Time to load utils op: 0.0008070468902587891 seconds 7: Time to load utils op: 0.0008053779602050781 seconds 7: Time to load utils op: 0.0010111331939697266 seconds 7: Time to load utils op: 0.0011973381042480469 secondsTime to load utils op: 0.001131296157836914 seconds 7: 7: Time to load utils op: 0.0011723041534423828 seconds 7: Time to load utils op: 0.001180887222290039 seconds 0: [2023-03-17 10:55:06,487] [INFO] [utils.py:827:see_memory_usage] before initializing group 0 0: [2023-03-17 10:55:06,487] [INFO] [utils.py:828:see_memory_usage] MA 0.28 GB Max_MA 0.28 GB CA 0.31 GB Max_CA 0 GB 0: [2023-03-17 10:55:06,488] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.48 GB, percent = 6.3% 0: [2023-03-17 10:55:06,603] [INFO] [utils.py:827:see_memory_usage] after initializing group 0 0: [2023-03-17 10:55:06,604] [INFO] [utils.py:828:see_memory_usage] MA 0.62 GB Max_MA 0.62 GB CA 0.82 GB Max_CA 1 GB 0: [2023-03-17 10:55:06,604] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.48 GB, percent = 6.3% 0: [2023-03-17 10:55:06,708] [INFO] [utils.py:827:see_memory_usage] before initializing group 1 0: [2023-03-17 10:55:06,709] [INFO] [utils.py:828:see_memory_usage] MA 0.62 GB Max_MA 0.62 GB CA 0.82 GB Max_CA 1 GB 0: [2023-03-17 10:55:06,709] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.48 GB, percent = 6.3% 0: [2023-03-17 10:55:06,813] [INFO] [utils.py:827:see_memory_usage] after initializing group 1 0: [2023-03-17 10:55:06,813] [INFO] [utils.py:828:see_memory_usage] MA 0.83 GB Max_MA 0.83 GB CA 1.13 GB Max_CA 1 GB 0: [2023-03-17 10:55:06,814] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.48 GB, percent = 6.3% 0: [2023-03-17 10:55:06,914] [INFO] [utils.py:827:see_memory_usage] before initializing group 2 0: [2023-03-17 10:55:06,914] [INFO] [utils.py:828:see_memory_usage] MA 0.83 GB Max_MA 0.83 GB CA 1.13 GB Max_CA 1 GB 0: [2023-03-17 10:55:06,914] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.48 GB, percent = 6.3% 0: [2023-03-17 10:55:07,016] [INFO] [utils.py:827:see_memory_usage] after initializing group 2 0: [2023-03-17 10:55:07,017] [INFO] [utils.py:828:see_memory_usage] MA 0.83 GB Max_MA 0.83 GB CA 1.13 GB Max_CA 1 GB 0: [2023-03-17 10:55:07,017] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.48 GB, percent = 6.3% 0: [2023-03-17 10:55:07,118] [INFO] [utils.py:827:see_memory_usage] before initialize_optimizer 0: [2023-03-17 10:55:07,118] [INFO] [utils.py:828:see_memory_usage] MA 0.83 GB Max_MA 0.83 GB CA 1.13 GB Max_CA 1 GB 0: [2023-03-17 10:55:07,119] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.48 GB, percent = 6.3% 0: [2023-03-17 10:55:07,225] [INFO] [utils.py:827:see_memory_usage] end initialize_optimizer 0: [2023-03-17 10:55:07,225] [INFO] [utils.py:828:see_memory_usage] MA 0.85 GB Max_MA 0.85 GB CA 1.13 GB Max_CA 1 GB 0: [2023-03-17 10:55:07,226] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.48 GB, percent = 6.3% 0: [2023-03-17 10:55:07,326] [INFO] [utils.py:827:see_memory_usage] end bf16_optimizer 0: [2023-03-17 10:55:07,327] [INFO] [utils.py:828:see_memory_usage] MA 0.85 GB Max_MA 0.85 GB CA 1.13 GB Max_CA 1 GB 0: [2023-03-17 10:55:07,327] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.48 GB, percent = 6.3% 0: [2023-03-17 10:55:07,327] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Final Optimizer = FusedAdam 0: [2023-03-17 10:55:07,327] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed using client LR scheduler 0: [2023-03-17 10:55:07,327] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed LR Scheduler = 0: [2023-03-17 10:55:07,327] [INFO] [logging.py:68:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0, 0.0, 0.0], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 0: [2023-03-17 10:55:07,327] [INFO] [config.py:1007:print] DeepSpeedEngine configuration: 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] activation_checkpointing_config { 0: "partition_activations": false, 0: "contiguous_memory_optimization": false, 0: "cpu_checkpointing": false, 0: "number_checkpoints": null, 0: "synchronize_checkpoint_boundary": false, 0: "profile": false 0: } 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True} 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] amp_enabled .................. False 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] amp_params ................... False 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] autotuning_config ............ { 0: "enabled": false, 0: "start_step": null, 0: "end_step": null, 0: "metric_path": null, 0: "arg_mappings": null, 0: "metric": "throughput", 0: "model_info": null, 0: "results_dir": "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/autotuning_results", 0: "exps_dir": "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/autotuning_exps", 0: "overwrite": true, 0: "fast": true, 0: "start_profile_step": 3, 0: "end_profile_step": 5, 0: "tuner_type": "gridsearch", 0: "tuner_early_stopping": 5, 0: "tuner_num_trials": 50, 0: "model_info_path": null, 0: "mp_size": 1, 0: "max_train_batch_size": null, 0: "min_train_batch_size": 1, 0: "max_train_micro_batch_size_per_gpu": 1.024000e+03, 0: "min_train_micro_batch_size_per_gpu": 1, 0: "num_tuning_micro_batch_sizes": 3 0: } 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] bfloat16_enabled ............. True 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] checkpoint_parallel_write_pipeline False 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] checkpoint_tag_validation_enabled True 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] checkpoint_tag_validation_fail False 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] comms_config ................. 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] communication_data_type ...... None 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_pa 0: rameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}} 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] curriculum_enabled ........... False 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] curriculum_params ............ False 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] dataloader_drop_last ......... False 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] disable_allgather ............ False 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] dump_state ................... False 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] dynamic_loss_scale_args ...... None 0: [2023-03-17 10:55:07,328] [INFO] [config.py:1011:print] eigenvalue_enabled ........... False 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] eigenvalue_gas_boundary_resolution 1 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] eigenvalue_layer_name ........ bert.encoder.layer 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] eigenvalue_layer_num ......... 0 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] eigenvalue_max_iter .......... 100 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] eigenvalue_stability ......... 1e-06 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] eigenvalue_tol ............... 0.01 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] eigenvalue_verbose ........... False 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] elasticity_enabled ........... False 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] flops_profiler_config ........ { 0: "enabled": false, 0: "profile_step": 1, 0: "module_depth": -1, 0: "top_modules": 1, 0: "detailed": true, 0: "output_file": null 0: } 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] fp16_auto_cast ............... None 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] fp16_enabled ................. False 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] fp16_master_weights_and_gradients False 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] global_rank .................. 0 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] gradient_accumulation_steps .. 1 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] gradient_clipping ............ 1.0 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] gradient_predivide_factor .... 1.0 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] initial_dynamic_scale ........ 1 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] load_universal_checkpoint .... False 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] loss_scale ................... 1.0 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] memory_breakdown ............. False 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] monitor_config ............... 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] nebula_config ................ { 0: "enabled": false, 0: "persistent_storage_path": null, 0: "persistent_time_interval": 100, 0: "num_of_version_in_retention": 2, 0: "enable_nebula_load": true, 0: "load_path": null 0: } 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] optimizer_legacy_fusion ...... False 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] optimizer_name ............... None 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] optimizer_params ............. None 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0} 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] pld_enabled .................. False 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] pld_params ................... False 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] prescale_gradients ........... False 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] scheduler_name ............... None 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] scheduler_params ............. None 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] sparse_attention ............. None 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] sparse_gradients_enabled ..... False 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] steps_per_print .............. 2000 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] train_batch_size ............. 256 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] train_micro_batch_size_per_gpu 4 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] use_node_local_storage ....... False 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] wall_clock_breakdown ......... False 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] world_size ................... 64 0: [2023-03-17 10:55:07,329] [INFO] [config.py:1011:print] zero_allow_untested_optimizer False 0: [2023-03-17 10:55:07,330] [INFO] [config.py:1011:print] zero_config .................. stage=0 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500000000 allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=False load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False 0: [2023-03-17 10:55:07,330] [INFO] [config.py:1011:print] zero_enabled ................. False 0: [2023-03-17 10:55:07,330] [INFO] [config.py:1011:print] zero_optimization_stage ...... 0 0: [2023-03-17 10:55:07,330] [INFO] [config.py:996:print_user_config] json = { 0: "train_micro_batch_size_per_gpu": 4, 0: "train_batch_size": 256, 0: "gradient_clipping": 1.0, 0: "zero_optimization": { 0: "stage": 0 0: }, 0: "bf16": { 0: "enabled": true 0: }, 0: "steps_per_print": 2.000000e+03, 0: "wall_clock_breakdown": false 0: } 0: Time to load utils op: 0.0004131793975830078 seconds 0: [2023-03-17 10:55:07,330] [INFO] [engine.py:87:__init__] CONFIG: micro_batches=1 micro_batch_size=4 0: [2023-03-17 10:55:07,341] [INFO] [engine.py:145:__init__] RANK=0 STAGE=0 LAYERS=22 [0, 22) STAGE_PARAMS=146525952 (146.526M) TOTAL_PARAMS=146525952 (146.526M) UNIQUE_PARAMS=146525952 (146.526M) 0: [2023-03-17 10:55:07,347] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 0: WARNING: could not find the metadata file checkpoints_146m91b100mdedup 0: will not load any checkpoints and will start from random 0: [2023-03-17 10:55:07,347] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 0: [2023-03-17 10:55:07,347] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 6: [2023-03-17 10:55:07,347] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 6: [2023-03-17 10:55:07,347] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 6: [2023-03-17 10:55:07,347] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 4: [2023-03-17 10:55:07,347] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 4: [2023-03-17 10:55:07,347] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 4: [2023-03-17 10:55:07,347] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 0: [2023-03-17 10:55:07,347] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 4: [2023-03-17 10:55:07,347] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 6: [2023-03-17 10:55:07,347] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: [2023-03-17 10:55:07,347] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: [2023-03-17 10:55:07,347] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 5: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 5: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 5: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 5: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 2: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 2: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 3: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 3: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 3: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 3: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 6: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 2: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 2: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 4: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 0: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 6: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 0: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 4: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 3: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 3: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 1: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 1: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 1: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 6: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 6: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 1: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 1: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 4: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 1: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 4: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 0: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 0: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 3: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 3: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 1: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 1: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 5: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 5: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 2: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 5: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 5: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 2: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 2: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: time (ms) | load-checkpoint: 7.21 2: [2023-03-17 10:55:07,348] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b100mdedup/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 0: estimated model parameters: 0.146525952 0: estimated model parameters without embeddings: 0.106319616 0: [after model, optimizer, and learning rate scheduler are built] datetime: 2023-03-17 10:55:08 0: > building train, validation, and test datasets ... 0: > datasets target sizes (minimum size): 0: train: 44416143 0: validation: 4608 0: test: 256 0: > building train, validation, and test datasets for GPT ... 0: > building dataset index ... 0: reading sizes... 0: reading pointers... 0: reading document index... 0: creating numpy buffer of mmap... 0: creating memory view of numpy buffer... 0: > finished creating indexed dataset in 0.009784 seconds 0: number of documents: 409500 0: > dataset split: 0: train: 0: document indices in [0, 409500) total of 409500 documents 0: > WARNING: could not find index map files, building the indices on rank 0 ... 0: > last epoch number of samples (45289) is smaller than 95.0% of number of samples per epoch (48281), setting separate_last_epoch to True 0: > elasped time to build and save doc-idx mapping (seconds): 32.196652 0: using: 0: number of documents: 409500 0: number of epochs: 920 0: sequence length: 2048 0: total number of samples: 44419136 0: > elasped time to build and save sample-idx mapping (seconds): 1.320620 0: > building shuffle index with split [0, 44370854) and [44370854, 44419136) ... 0: > elasped time to build and save shuffle-idx mapping (seconds): 3.346765 0: > loading doc-idx mapping from /scratch/project_462000119/data/c4_subsampled/gpt2tok_c4_en_dedup_100M_text_document_train_indexmap_44416143ns_2048sl_1234s_doc_idx.npy 0: > loading sample-idx mapping from /scratch/project_462000119/data/c4_subsampled/gpt2tok_c4_en_dedup_100M_text_document_train_indexmap_44416143ns_2048sl_1234s_sample_idx.npy 0: > loading shuffle-idx mapping from /scratch/project_462000119/data/c4_subsampled/gpt2tok_c4_en_dedup_100M_text_document_train_indexmap_44416143ns_2048sl_1234s_shuffle_idx.npy 0: loaded indexed file in 0.051 seconds 0: total number of samples: 44419137 0: total number of epochs: 920 0: > building dataset index ... 0: reading sizes... 0: reading pointers... 0: reading document index... 0: creating numpy buffer of mmap... 0: creating memory view of numpy buffer... 0: > finished creating indexed dataset in 0.021993 seconds 0: number of documents: 364608 0: > dataset split: 0: validation: 0: document indices in [0, 364608) total of 364608 documents 0: > loading doc-idx mapping from /scratch/project_462000119/data/c4_validation/gpt2tok_c4validation_rerun_text_document_validation_indexmap_4608ns_2048sl_1234s_doc_idx.npy 0: > loading sample-idx mapping from /scratch/project_462000119/data/c4_validation/gpt2tok_c4validation_rerun_text_document_validation_indexmap_4608ns_2048sl_1234s_sample_idx.npy 0: > loading shuffle-idx mapping from /scratch/project_462000119/data/c4_validation/gpt2tok_c4validation_rerun_text_document_validation_indexmap_4608ns_2048sl_1234s_shuffle_idx.npy 0: loaded indexed file in 0.074 seconds 0: total number of samples: 84978 0: total number of epochs: 1 0: > finished creating GPT datasets ... 0: [after dataloaders are built] datetime: 2023-03-17 10:55:59 0: done with setup ... 0: training ... 0: Number of parameters: [tensor rank - pipeline rank] w/ and w/o embeddings: 7: time (ms) | model-and-optimizer-setup: 17485.39 | train/valid/test-data-iterators-setup: 49805.82 0: [000-000] 0.1465B / 0.1063B 0: [before the start of training step] datetime: 2023-03-17 10:55:59 0: [2023-03-17 10:56:00,272] [INFO] [checkpointing.py:553:forward] Activation Checkpointing Information 0: [2023-03-17 10:56:00,273] [INFO] [checkpointing.py:554:forward] ----Partition Activations False, CPU CHECKPOINTING False 0: [2023-03-17 10:56:00,273] [INFO] [checkpointing.py:557:forward] ----contiguous Memory Checkpointing False with None total layers 0: [2023-03-17 10:56:00,273] [INFO] [checkpointing.py:560:forward] ----Synchronization False 0: [2023-03-17 10:56:00,273] [INFO] [checkpointing.py:561:forward] ----Profiling time in checkpointing False 0: [Rank 0] (after 100 iterations) memory (MB) | allocated: 2729.60986328125 | max allocated: 5303.921875 | reserved: 6818.0 | max reserved: 6818.0 7: iteration 100/ 173500 | consumed samples: 25600 | consumed tokens: 52428800 | elapsed time per iteration (s): 0.49 | learning rate: 1.153E-05 | global batch size: 256 | lm loss: 9.615845E+00 | grad norm: 1.531 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 525.722 | TFLOPs: 24.54 | 7: iteration 200/ 173500 | consumed samples: 51200 | consumed tokens: 104857600 | elapsed time per iteration (s): 0.39 | learning rate: 2.305E-05 | global batch size: 256 | lm loss: 7.946847E+00 | grad norm: 0.887 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 657.805 | TFLOPs: 30.70 | 7: iteration 300/ 173500 | consumed samples: 76800 | consumed tokens: 157286400 | elapsed time per iteration (s): 0.38 | learning rate: 3.458E-05 | global batch size: 256 | lm loss: 6.915150E+00 | grad norm: 1.044 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 671.397 | TFLOPs: 31.34 | 7: iteration 400/ 173500 | consumed samples: 102400 | consumed tokens: 209715200 | elapsed time per iteration (s): 0.39 | learning rate: 4.611E-05 | global batch size: 256 | lm loss: 6.500023E+00 | grad norm: 0.624 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 664.595 | TFLOPs: 31.02 | 7: iteration 500/ 173500 | consumed samples: 128000 | consumed tokens: 262144000 | elapsed time per iteration (s): 0.38 | learning rate: 5.764E-05 | global batch size: 256 | lm loss: 6.293899E+00 | grad norm: 0.852 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 672.453 | TFLOPs: 31.39 | 7: iteration 600/ 173500 | consumed samples: 153600 | consumed tokens: 314572800 | elapsed time per iteration (s): 0.38 | learning rate: 6.916E-05 | global batch size: 256 | lm loss: 6.147126E+00 | grad norm: 0.943 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 674.986 | TFLOPs: 31.51 | 7: iteration 700/ 173500 | consumed samples: 179200 | consumed tokens: 367001600 | elapsed time per iteration (s): 0.38 | learning rate: 8.069E-05 | global batch size: 256 | lm loss: 6.009447E+00 | grad norm: 0.989 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 671.702 | TFLOPs: 31.35 | 7: iteration 800/ 173500 | consumed samples: 204800 | consumed tokens: 419430400 | elapsed time per iteration (s): 0.38 | learning rate: 9.222E-05 | global batch size: 256 | lm loss: 5.853611E+00 | grad norm: 1.588 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 675.217 | TFLOPs: 31.52 | 7: iteration 900/ 173500 | consumed samples: 230400 | consumed tokens: 471859200 | elapsed time per iteration (s): 0.38 | learning rate: 1.037E-04 | global batch size: 256 | lm loss: 5.705782E+00 | grad norm: 1.477 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 676.078 | TFLOPs: 31.56 | 7: iteration 1000/ 173500 | consumed samples: 256000 | consumed tokens: 524288000 | elapsed time per iteration (s): 0.38 | learning rate: 1.153E-04 | global batch size: 256 | lm loss: 5.565979E+00 | grad norm: 0.726 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 671.986 | TFLOPs: 31.37 | 7: iteration 1100/ 173500 | consumed samples: 281600 | consumed tokens: 576716800 | elapsed time per iteration (s): 0.38 | learning rate: 1.268E-04 | global batch size: 256 | lm loss: 5.421600E+00 | grad norm: 1.092 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 670.266 | TFLOPs: 31.29 | 7: iteration 1200/ 173500 | consumed samples: 307200 | consumed tokens: 629145600 | elapsed time per iteration (s): 0.38 | learning rate: 1.383E-04 | global batch size: 256 | lm loss: 5.289799E+00 | grad norm: 0.832 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 672.397 | TFLOPs: 31.39 | 7: iteration 1300/ 173500 | consumed samples: 332800 | consumed tokens: 681574400 | elapsed time per iteration (s): 0.38 | learning rate: 1.499E-04 | global batch size: 256 | lm loss: 5.170934E+00 | grad norm: 1.176 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 665.016 | TFLOPs: 31.04 | 7: iteration 1400/ 173500 | consumed samples: 358400 | consumed tokens: 734003200 | elapsed time per iteration (s): 0.39 | learning rate: 1.614E-04 | global batch size: 256 | lm loss: 5.068483E+00 | grad norm: 0.799 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 651.025 | TFLOPs: 30.39 | 7: iteration 1500/ 173500 | consumed samples: 384000 | consumed tokens: 786432000 | elapsed time per iteration (s): 0.38 | learning rate: 1.729E-04 | global batch size: 256 | lm loss: 4.965949E+00 | grad norm: 0.935 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 670.684 | TFLOPs: 31.31 | 7: iteration 1600/ 173500 | consumed samples: 409600 | consumed tokens: 838860800 | elapsed time per iteration (s): 0.38 | learning rate: 1.844E-04 | global batch size: 256 | lm loss: 4.870196E+00 | grad norm: 0.479 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 674.165 | TFLOPs: 31.47 | 7: iteration 1700/ 173500 | consumed samples: 435200 | consumed tokens: 891289600 | elapsed time per iteration (s): 0.38 | learning rate: 1.960E-04 | global batch size: 256 | lm loss: 4.784067E+00 | grad norm: 0.610 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 677.196 | TFLOPs: 31.61 | 7: iteration 1800/ 173500 | consumed samples: 460800 | consumed tokens: 943718400 | elapsed time per iteration (s): 0.38 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.714541E+00 | grad norm: 0.415 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 671.437 | TFLOPs: 31.34 | 7: iteration 1900/ 173500 | consumed samples: 486400 | consumed tokens: 996147200 | elapsed time per iteration (s): 0.39 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.647032E+00 | grad norm: 0.565 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 663.983 | TFLOPs: 30.99 | 0: [2023-03-17 11:08:54,670] [INFO] [logging.py:68:log_dist] [Rank 0] step=2000, skipped=0, lr=[0.00019999894289482022, 0.00019999894289482022, 0.00019999894289482022], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 2000/ 173500 | consumed samples: 512000 | consumed tokens: 1048576000 | elapsed time per iteration (s): 0.38 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.595067E+00 | grad norm: 0.491 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 666.859 | TFLOPs: 31.13 | 0: steps: 2000 loss: 4.5769 iter time (s): 0.386 samples/sec: 663.766 7: iteration 2100/ 173500 | consumed samples: 537600 | consumed tokens: 1101004800 | elapsed time per iteration (s): 0.38 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.547905E+00 | grad norm: 0.595 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 668.625 | TFLOPs: 31.21 | 7: iteration 2200/ 173500 | consumed samples: 563200 | consumed tokens: 1153433600 | elapsed time per iteration (s): 0.38 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.509464E+00 | grad norm: 0.459 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 669.143 | TFLOPs: 31.23 | 7: iteration 2300/ 173500 | consumed samples: 588800 | consumed tokens: 1205862400 | elapsed time per iteration (s): 0.38 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.473263E+00 | grad norm: 0.509 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 680.356 | TFLOPs: 31.76 | 7: iteration 2400/ 173500 | consumed samples: 614400 | consumed tokens: 1258291200 | elapsed time per iteration (s): 0.38 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.441313E+00 | grad norm: 0.519 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 666.463 | TFLOPs: 31.11 | 7: iteration 2500/ 173500 | consumed samples: 640000 | consumed tokens: 1310720000 | elapsed time per iteration (s): 0.38 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.413993E+00 | grad norm: 0.419 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 680.433 | TFLOPs: 31.76 | 7: iteration 2600/ 173500 | consumed samples: 665600 | consumed tokens: 1363148800 | elapsed time per iteration (s): 0.38 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.387156E+00 | grad norm: 0.366 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 673.592 | TFLOPs: 31.44 | 7: iteration 2700/ 173500 | consumed samples: 691200 | consumed tokens: 1415577600 | elapsed time per iteration (s): 0.38 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.356254E+00 | grad norm: 0.400 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 674.217 | TFLOPs: 31.47 | 7: iteration 2800/ 173500 | consumed samples: 716800 | consumed tokens: 1468006400 | elapsed time per iteration (s): 0.38 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.334037E+00 | grad norm: 0.514 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 680.126 | TFLOPs: 31.75 | 7: iteration 2900/ 173500 | consumed samples: 742400 | consumed tokens: 1520435200 | elapsed time per iteration (s): 0.38 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.311415E+00 | grad norm: 0.434 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 673.574 | TFLOPs: 31.44 | 7: iteration 3000/ 173500 | consumed samples: 768000 | consumed tokens: 1572864000 | elapsed time per iteration (s): 0.38 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.289158E+00 | grad norm: 0.408 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 681.437 | TFLOPs: 31.81 | 7: iteration 3100/ 173500 | consumed samples: 793600 | consumed tokens: 1625292800 | elapsed time per iteration (s): 0.38 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.269532E+00 | grad norm: 0.449 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 675.889 | TFLOPs: 31.55 | 7: iteration 3200/ 173500 | consumed samples: 819200 | consumed tokens: 1677721600 | elapsed time per iteration (s): 0.38 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.250038E+00 | grad norm: 0.406 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 682.221 | TFLOPs: 31.84 | 7: iteration 3300/ 173500 | consumed samples: 844800 | consumed tokens: 1730150400 | elapsed time per iteration (s): 0.38 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.230816E+00 | grad norm: 0.356 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 682.354 | TFLOPs: 31.85 | 7: iteration 3400/ 173500 | consumed samples: 870400 | consumed tokens: 1782579200 | elapsed time per iteration (s): 0.38 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.213364E+00 | grad norm: 0.435 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 681.111 | TFLOPs: 31.79 | 7: iteration 3500/ 173500 | consumed samples: 896000 | consumed tokens: 1835008000 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.199569E+00 | grad norm: 0.437 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 684.710 | TFLOPs: 31.96 | 7: iteration 3600/ 173500 | consumed samples: 921600 | consumed tokens: 1887436800 | elapsed time per iteration (s): 0.38 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.182302E+00 | grad norm: 0.390 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 678.844 | TFLOPs: 31.69 | 7: iteration 3700/ 173500 | consumed samples: 947200 | consumed tokens: 1939865600 | elapsed time per iteration (s): 0.38 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.164708E+00 | grad norm: 0.347 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 671.475 | TFLOPs: 31.34 | 7: iteration 3800/ 173500 | consumed samples: 972800 | consumed tokens: 1992294400 | elapsed time per iteration (s): 0.38 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.151073E+00 | grad norm: 0.454 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 682.205 | TFLOPs: 31.84 | 7: iteration 3900/ 173500 | consumed samples: 998400 | consumed tokens: 2044723200 | elapsed time per iteration (s): 0.38 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.137338E+00 | grad norm: 0.358 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 679.950 | TFLOPs: 31.74 | 0: [2023-03-17 11:21:30,360] [INFO] [logging.py:68:log_dist] [Rank 0] step=4000, skipped=0, lr=[0.00019992278300259638, 0.00019992278300259638, 0.00019992278300259638], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 4000/ 173500 | consumed samples: 1024000 | consumed tokens: 2097152000 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.125865E+00 | grad norm: 0.337 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 684.888 | TFLOPs: 31.97 | 0: steps: 4000 loss: 4.1055 iter time (s): 0.376 samples/sec: 681.270 7: iteration 4100/ 173500 | consumed samples: 1049600 | consumed tokens: 2149580800 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.110681E+00 | grad norm: 0.439 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 683.226 | TFLOPs: 31.89 | 7: iteration 4200/ 173500 | consumed samples: 1075200 | consumed tokens: 2202009600 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.098109E+00 | grad norm: 0.398 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 685.538 | TFLOPs: 32.00 | 7: iteration 4300/ 173500 | consumed samples: 1100800 | consumed tokens: 2254438400 | elapsed time per iteration (s): 0.38 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.084235E+00 | grad norm: 0.407 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 682.282 | TFLOPs: 31.85 | 7: iteration 4400/ 173500 | consumed samples: 1126400 | consumed tokens: 2306867200 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.071701E+00 | grad norm: 0.327 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 683.765 | TFLOPs: 31.92 | 7: iteration 4500/ 173500 | consumed samples: 1152000 | consumed tokens: 2359296000 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.060500E+00 | grad norm: 0.385 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 683.082 | TFLOPs: 31.88 | 7: iteration 4600/ 173500 | consumed samples: 1177600 | consumed tokens: 2411724800 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.050145E+00 | grad norm: 0.344 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.471 | TFLOPs: 32.04 | 7: iteration 4700/ 173500 | consumed samples: 1203200 | consumed tokens: 2464153600 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.039188E+00 | grad norm: 0.404 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 685.307 | TFLOPs: 31.99 | 7: iteration 4800/ 173500 | consumed samples: 1228800 | consumed tokens: 2516582400 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.027598E+00 | grad norm: 0.400 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 684.149 | TFLOPs: 31.93 | 7: iteration 4900/ 173500 | consumed samples: 1254400 | consumed tokens: 2569011200 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.019306E+00 | grad norm: 0.342 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 685.965 | TFLOPs: 32.02 | 7: iteration 5000/ 173500 | consumed samples: 1280000 | consumed tokens: 2621440000 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.009735E+00 | grad norm: 0.359 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.195 | TFLOPs: 32.03 | 7: iteration 5100/ 173500 | consumed samples: 1305600 | consumed tokens: 2673868800 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 3.998331E+00 | grad norm: 0.331 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.333 | TFLOPs: 32.04 | 7: iteration 5200/ 173500 | consumed samples: 1331200 | consumed tokens: 2726297600 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 3.992051E+00 | grad norm: 0.331 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.524 | TFLOPs: 32.09 | 7: iteration 5300/ 173500 | consumed samples: 1356800 | consumed tokens: 2778726400 | elapsed time per iteration (s): 0.39 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 3.982103E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 657.160 | TFLOPs: 30.67 | 7: iteration 5400/ 173500 | consumed samples: 1382400 | consumed tokens: 2831155200 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 3.972947E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 685.645 | TFLOPs: 32.00 | 7: iteration 5500/ 173500 | consumed samples: 1408000 | consumed tokens: 2883584000 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 3.963240E+00 | grad norm: 0.351 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 690.137 | TFLOPs: 32.21 | 7: iteration 5600/ 173500 | consumed samples: 1433600 | consumed tokens: 2936012800 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 3.957315E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 683.103 | TFLOPs: 31.88 | 7: iteration 5700/ 173500 | consumed samples: 1459200 | consumed tokens: 2988441600 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 3.950648E+00 | grad norm: 0.326 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.129 | TFLOPs: 32.03 | 7: iteration 5800/ 173500 | consumed samples: 1484800 | consumed tokens: 3040870400 | elapsed time per iteration (s): 0.38 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 3.940122E+00 | grad norm: 0.337 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 682.359 | TFLOPs: 31.85 | 7: iteration 5900/ 173500 | consumed samples: 1510400 | consumed tokens: 3093299200 | elapsed time per iteration (s): 0.38 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 3.932303E+00 | grad norm: 0.342 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 678.329 | TFLOPs: 31.66 | 0: [2023-03-17 11:33:59,526] [INFO] [logging.py:68:log_dist] [Rank 0] step=6000, skipped=0, lr=[0.0001997263111243839, 0.0001997263111243839, 0.0001997263111243839], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 6000/ 173500 | consumed samples: 1536000 | consumed tokens: 3145728000 | elapsed time per iteration (s): 0.37 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 3.923311E+00 | grad norm: 0.347 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.119 | TFLOPs: 32.07 | 0: steps: 6000 loss: 3.9008 iter time (s): 0.373 samples/sec: 687.043 7: iteration 6100/ 173500 | consumed samples: 1561600 | consumed tokens: 3198156800 | elapsed time per iteration (s): 0.37 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 3.917387E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 682.781 | TFLOPs: 31.87 | 7: iteration 6200/ 173500 | consumed samples: 1587200 | consumed tokens: 3250585600 | elapsed time per iteration (s): 0.38 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 3.911739E+00 | grad norm: 0.339 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 682.350 | TFLOPs: 31.85 | 7: iteration 6300/ 173500 | consumed samples: 1612800 | consumed tokens: 3303014400 | elapsed time per iteration (s): 0.37 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 3.903886E+00 | grad norm: 0.416 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 683.675 | TFLOPs: 31.91 | 7: iteration 6400/ 173500 | consumed samples: 1638400 | consumed tokens: 3355443200 | elapsed time per iteration (s): 0.37 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 3.898456E+00 | grad norm: 0.332 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 683.271 | TFLOPs: 31.89 | 7: iteration 6500/ 173500 | consumed samples: 1664000 | consumed tokens: 3407872000 | elapsed time per iteration (s): 0.37 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 3.889631E+00 | grad norm: 0.345 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.452 | TFLOPs: 32.09 | 7: iteration 6600/ 173500 | consumed samples: 1689600 | consumed tokens: 3460300800 | elapsed time per iteration (s): 0.38 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.882668E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 682.436 | TFLOPs: 31.85 | 7: iteration 6700/ 173500 | consumed samples: 1715200 | consumed tokens: 3512729600 | elapsed time per iteration (s): 0.37 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.875456E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.648 | TFLOPs: 32.14 | 7: iteration 6800/ 173500 | consumed samples: 1740800 | consumed tokens: 3565158400 | elapsed time per iteration (s): 0.37 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.871413E+00 | grad norm: 0.278 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.976 | TFLOPs: 32.07 | 7: iteration 6900/ 173500 | consumed samples: 1766400 | consumed tokens: 3617587200 | elapsed time per iteration (s): 0.37 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.861951E+00 | grad norm: 0.342 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.873 | TFLOPs: 32.11 | 7: iteration 7000/ 173500 | consumed samples: 1792000 | consumed tokens: 3670016000 | elapsed time per iteration (s): 0.38 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.860483E+00 | grad norm: 0.281 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 665.515 | TFLOPs: 31.06 | 7: iteration 7100/ 173500 | consumed samples: 1817600 | consumed tokens: 3722444800 | elapsed time per iteration (s): 0.37 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.853114E+00 | grad norm: 0.371 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 690.113 | TFLOPs: 32.21 | 7: iteration 7200/ 173500 | consumed samples: 1843200 | consumed tokens: 3774873600 | elapsed time per iteration (s): 0.37 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.848189E+00 | grad norm: 0.345 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.717 | TFLOPs: 32.15 | 7: iteration 7300/ 173500 | consumed samples: 1868800 | consumed tokens: 3827302400 | elapsed time per iteration (s): 0.37 | learning rate: 1.995E-04 | global batch size: 256 | lm loss: 3.840704E+00 | grad norm: 0.342 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 685.760 | TFLOPs: 32.01 | 7: iteration 7400/ 173500 | consumed samples: 1894400 | consumed tokens: 3879731200 | elapsed time per iteration (s): 0.37 | learning rate: 1.995E-04 | global batch size: 256 | lm loss: 3.835959E+00 | grad norm: 0.272 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 689.300 | TFLOPs: 32.17 | 7: iteration 7500/ 173500 | consumed samples: 1920000 | consumed tokens: 3932160000 | elapsed time per iteration (s): 0.37 | learning rate: 1.995E-04 | global batch size: 256 | lm loss: 3.831573E+00 | grad norm: 0.267 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.798 | TFLOPs: 32.06 | 7: iteration 7600/ 173500 | consumed samples: 1945600 | consumed tokens: 3984588800 | elapsed time per iteration (s): 0.37 | learning rate: 1.995E-04 | global batch size: 256 | lm loss: 3.825728E+00 | grad norm: 0.320 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 693.016 | TFLOPs: 32.35 | 7: iteration 7700/ 173500 | consumed samples: 1971200 | consumed tokens: 4037017600 | elapsed time per iteration (s): 0.39 | learning rate: 1.995E-04 | global batch size: 256 | lm loss: 3.819348E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 661.954 | TFLOPs: 30.90 | 7: iteration 7800/ 173500 | consumed samples: 1996800 | consumed tokens: 4089446400 | elapsed time per iteration (s): 0.37 | learning rate: 1.994E-04 | global batch size: 256 | lm loss: 3.815590E+00 | grad norm: 0.326 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.955 | TFLOPs: 32.11 | 7: iteration 7900/ 173500 | consumed samples: 2022400 | consumed tokens: 4141875200 | elapsed time per iteration (s): 0.37 | learning rate: 1.994E-04 | global batch size: 256 | lm loss: 3.810998E+00 | grad norm: 0.270 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 684.337 | TFLOPs: 31.94 | 0: [2023-03-17 11:46:27,813] [INFO] [logging.py:68:log_dist] [Rank 0] step=8000, skipped=0, lr=[0.00019940979012929202, 0.00019940979012929202, 0.00019940979012929202], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 8000/ 173500 | consumed samples: 2048000 | consumed tokens: 4194304000 | elapsed time per iteration (s): 0.37 | learning rate: 1.994E-04 | global batch size: 256 | lm loss: 3.804476E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.282 | TFLOPs: 32.08 | 0: steps: 8000 loss: 3.8164 iter time (s): 0.372 samples/sec: 688.161 7: iteration 8100/ 173500 | consumed samples: 2073600 | consumed tokens: 4246732800 | elapsed time per iteration (s): 0.39 | learning rate: 1.994E-04 | global batch size: 256 | lm loss: 3.801992E+00 | grad norm: 0.330 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 663.987 | TFLOPs: 30.99 | 7: iteration 8200/ 173500 | consumed samples: 2099200 | consumed tokens: 4299161600 | elapsed time per iteration (s): 0.37 | learning rate: 1.994E-04 | global batch size: 256 | lm loss: 3.796355E+00 | grad norm: 0.331 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 689.281 | TFLOPs: 32.17 | 7: iteration 8300/ 173500 | consumed samples: 2124800 | consumed tokens: 4351590400 | elapsed time per iteration (s): 0.37 | learning rate: 1.994E-04 | global batch size: 256 | lm loss: 3.789505E+00 | grad norm: 0.374 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 684.268 | TFLOPs: 31.94 | 7: iteration 8400/ 173500 | consumed samples: 2150400 | consumed tokens: 4404019200 | elapsed time per iteration (s): 0.37 | learning rate: 1.993E-04 | global batch size: 256 | lm loss: 3.785540E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.772 | TFLOPs: 32.06 | 7: iteration 8500/ 173500 | consumed samples: 2176000 | consumed tokens: 4456448000 | elapsed time per iteration (s): 0.37 | learning rate: 1.993E-04 | global batch size: 256 | lm loss: 3.783622E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.562 | TFLOPs: 32.09 | 7: iteration 8600/ 173500 | consumed samples: 2201600 | consumed tokens: 4508876800 | elapsed time per iteration (s): 0.37 | learning rate: 1.993E-04 | global batch size: 256 | lm loss: 3.779001E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.616 | TFLOPs: 32.14 | 7: iteration 8700/ 173500 | consumed samples: 2227200 | consumed tokens: 4561305600 | elapsed time per iteration (s): 0.37 | learning rate: 1.993E-04 | global batch size: 256 | lm loss: 3.771870E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 689.347 | TFLOPs: 32.18 | 7: iteration 8800/ 173500 | consumed samples: 2252800 | consumed tokens: 4613734400 | elapsed time per iteration (s): 0.37 | learning rate: 1.992E-04 | global batch size: 256 | lm loss: 3.771353E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.707 | TFLOPs: 32.05 | 7: iteration 8900/ 173500 | consumed samples: 2278400 | consumed tokens: 4666163200 | elapsed time per iteration (s): 0.37 | learning rate: 1.992E-04 | global batch size: 256 | lm loss: 3.765146E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.321 | TFLOPs: 32.13 | 7: iteration 9000/ 173500 | consumed samples: 2304000 | consumed tokens: 4718592000 | elapsed time per iteration (s): 0.37 | learning rate: 1.992E-04 | global batch size: 256 | lm loss: 3.761529E+00 | grad norm: 0.343 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 689.080 | TFLOPs: 32.16 | 7: iteration 9100/ 173500 | consumed samples: 2329600 | consumed tokens: 4771020800 | elapsed time per iteration (s): 0.37 | learning rate: 1.992E-04 | global batch size: 256 | lm loss: 3.756257E+00 | grad norm: 0.346 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.931 | TFLOPs: 32.06 | 7: iteration 9200/ 173500 | consumed samples: 2355200 | consumed tokens: 4823449600 | elapsed time per iteration (s): 0.37 | learning rate: 1.992E-04 | global batch size: 256 | lm loss: 3.753001E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 685.918 | TFLOPs: 32.02 | 7: iteration 9300/ 173500 | consumed samples: 2380800 | consumed tokens: 4875878400 | elapsed time per iteration (s): 0.37 | learning rate: 1.991E-04 | global batch size: 256 | lm loss: 3.748097E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.031 | TFLOPs: 32.11 | 7: iteration 9400/ 173500 | consumed samples: 2406400 | consumed tokens: 4928307200 | elapsed time per iteration (s): 0.37 | learning rate: 1.991E-04 | global batch size: 256 | lm loss: 3.743990E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.581 | TFLOPs: 32.09 | 7: iteration 9500/ 173500 | consumed samples: 2432000 | consumed tokens: 4980736000 | elapsed time per iteration (s): 0.37 | learning rate: 1.991E-04 | global batch size: 256 | lm loss: 3.740213E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.278 | TFLOPs: 32.03 | 7: iteration 9600/ 173500 | consumed samples: 2457600 | consumed tokens: 5033164800 | elapsed time per iteration (s): 0.37 | learning rate: 1.991E-04 | global batch size: 256 | lm loss: 3.735660E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.310 | TFLOPs: 32.03 | 7: iteration 9700/ 173500 | consumed samples: 2483200 | consumed tokens: 5085593600 | elapsed time per iteration (s): 0.37 | learning rate: 1.990E-04 | global batch size: 256 | lm loss: 3.733908E+00 | grad norm: 0.335 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 684.043 | TFLOPs: 31.93 | 7: iteration 9800/ 173500 | consumed samples: 2508800 | consumed tokens: 5138022400 | elapsed time per iteration (s): 0.38 | learning rate: 1.990E-04 | global batch size: 256 | lm loss: 3.729467E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 675.423 | TFLOPs: 31.53 | 7: iteration 9900/ 173500 | consumed samples: 2534400 | consumed tokens: 5190451200 | elapsed time per iteration (s): 0.37 | learning rate: 1.990E-04 | global batch size: 256 | lm loss: 3.727076E+00 | grad norm: 0.286 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 689.696 | TFLOPs: 32.19 | 0: [2023-03-17 11:58:54,555] [INFO] [logging.py:68:log_dist] [Rank 0] step=10000, skipped=0, lr=[0.00019897364350587667, 0.00019897364350587667, 0.00019897364350587667], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 10000/ 173500 | consumed samples: 2560000 | consumed tokens: 5242880000 | elapsed time per iteration (s): 0.37 | learning rate: 1.990E-04 | global batch size: 256 | lm loss: 3.723191E+00 | grad norm: 0.281 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 689.823 | TFLOPs: 32.20 | 0: steps: 10000 loss: 3.7144 iter time (s): 0.371 samples/sec: 689.726 7: ------------------------------------------------------------------------------------------------ 7: validation loss at iteration 10000 | lm loss value: 3.921669E+00 | lm loss PPL: 5.048463E+01 | 7: ------------------------------------------------------------------------------------------------ 0: saving checkpoint at iteration 10000 to checkpoints_146m91b100mdedup 0: [2023-03-17 11:58:54,677] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step10000 is begin to save! 0: [2023-03-17 11:58:55,168] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_01-model_00-model_states.pt... 0: [2023-03-17 11:58:55,299] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_01-model_00-model_states.pt. 0: [2023-03-17 11:58:55,299] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_03-model_00-model_states.pt... 0: [2023-03-17 11:58:55,316] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_03-model_00-model_states.pt. 0: [2023-03-17 11:58:55,316] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_04-model_00-model_states.pt... 0: [2023-03-17 11:58:55,332] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_04-model_00-model_states.pt. 0: [2023-03-17 11:58:55,332] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_05-model_00-model_states.pt... 0: [2023-03-17 11:58:55,349] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_05-model_00-model_states.pt. 0: [2023-03-17 11:58:55,349] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_06-model_00-model_states.pt... 0: [2023-03-17 11:58:55,364] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_06-model_00-model_states.pt. 0: [2023-03-17 11:58:55,365] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_07-model_00-model_states.pt... 0: [2023-03-17 11:58:55,381] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_07-model_00-model_states.pt. 0: [2023-03-17 11:58:55,381] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_08-model_00-model_states.pt... 0: [2023-03-17 11:58:55,397] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_08-model_00-model_states.pt. 0: [2023-03-17 11:58:55,397] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_09-model_00-model_states.pt... 0: [2023-03-17 11:58:55,413] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_09-model_00-model_states.pt. 0: [2023-03-17 11:58:55,413] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_10-model_00-model_states.pt... 0: [2023-03-17 11:58:55,429] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_10-model_00-model_states.pt. 0: [2023-03-17 11:58:55,429] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_11-model_00-model_states.pt... 0: [2023-03-17 11:58:55,445] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_11-model_00-model_states.pt. 0: [2023-03-17 11:58:55,445] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_12-model_00-model_states.pt... 0: [2023-03-17 11:58:55,461] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_12-model_00-model_states.pt. 0: [2023-03-17 11:58:55,461] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_13-model_00-model_states.pt... 0: [2023-03-17 11:58:55,478] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_13-model_00-model_states.pt. 0: [2023-03-17 11:58:55,478] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_14-model_00-model_states.pt... 0: [2023-03-17 11:58:55,493] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_14-model_00-model_states.pt. 0: [2023-03-17 11:58:55,494] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_15-model_00-model_states.pt... 0: [2023-03-17 11:58:55,509] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_15-model_00-model_states.pt. 0: [2023-03-17 11:58:55,510] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_16-model_00-model_states.pt... 0: [2023-03-17 11:58:55,525] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_16-model_00-model_states.pt. 0: [2023-03-17 11:58:55,526] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_17-model_00-model_states.pt... 0: [2023-03-17 11:58:55,541] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_17-model_00-model_states.pt. 0: [2023-03-17 11:58:55,541] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/layer_19-model_00-model_states.pt... 0: [2023-03-17 11:58:55,543] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/layer_19-model_00-model_states.pt. 0: [2023-03-17 11:58:55,543] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_146m91b100mdedup/global_step10000/mp_rank_00_model_states.pt 0: [2023-03-17 11:58:55,543] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/mp_rank_00_model_states.pt... 0: [2023-03-17 11:58:55,547] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/mp_rank_00_model_states.pt. 0: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:58:55,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:58:55,607] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:58:55,607] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:58:55,607] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:58:55,607] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt 0: [2023-03-17 11:58:55,607] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt 0: [2023-03-17 11:58:55,607] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: [2023-03-17 11:58:55,607] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt 0: [2023-03-17 11:58:55,607] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: [2023-03-17 11:58:55,607] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: [2023-03-17 11:58:55,608] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:58:55,608] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt 0: [2023-03-17 11:58:55,608] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: [2023-03-17 11:58:55,613] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:58:55,614] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt 0: [2023-03-17 11:58:55,614] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: [2023-03-17 11:58:55,614] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:58:55,615] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:58:55,615] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt 0: [2023-03-17 11:58:55,615] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 11:58:55,615] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:58:55,615] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt 4: [2023-03-17 11:58:55,615] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: [2023-03-17 11:58:55,616] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:58:55,616] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt 0: [2023-03-17 11:58:55,616] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 11:58:55,616] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:58:55,616] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt 4: [2023-03-17 11:58:55,616] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 11:58:55,616] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:58:55,616] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt 4: [2023-03-17 11:58:55,616] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 3: [2023-03-17 11:58:55,616] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:58:55,617] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt 3: [2023-03-17 11:58:55,617] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 3: [2023-03-17 11:58:55,617] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:58:55,617] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt 3: [2023-03-17 11:58:55,617] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 3: [2023-03-17 11:58:55,618] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:58:55,618] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:58:55,618] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:58:55,618] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt 3: [2023-03-17 11:58:55,618] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt 3: [2023-03-17 11:58:55,618] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt 3: [2023-03-17 11:58:55,618] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 3: [2023-03-17 11:58:55,618] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 3: [2023-03-17 11:58:55,618] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 3: [2023-03-17 11:58:55,618] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:58:55,618] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:58:55,618] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:58:55,618] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt 3: [2023-03-17 11:58:55,618] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt 3: [2023-03-17 11:58:55,618] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 3: [2023-03-17 11:58:55,618] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt 3: [2023-03-17 11:58:55,618] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 3: [2023-03-17 11:58:55,618] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 11:58:55,621] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:58:55,621] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:58:55,621] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:58:55,621] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:58:55,621] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:58:55,621] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:58:55,621] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:58:55,621] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt 2: [2023-03-17 11:58:55,621] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt 2: [2023-03-17 11:58:55,621] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt 2: [2023-03-17 11:58:55,621] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt 2: [2023-03-17 11:58:55,621] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt 2: [2023-03-17 11:58:55,621] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt 2: [2023-03-17 11:58:55,621] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt 2: [2023-03-17 11:58:55,621] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 11:58:55,621] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 11:58:55,621] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 11:58:55,621] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 11:58:55,621] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 11:58:55,621] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 11:58:55,621] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 11:58:55,621] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:58:55,621] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt 2: [2023-03-17 11:58:55,621] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 5: [2023-03-17 11:58:55,625] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:58:55,625] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt 5: [2023-03-17 11:58:55,625] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 5: [2023-03-17 11:58:55,625] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:58:55,626] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt 5: [2023-03-17 11:58:55,626] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 11:58:55,627] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:58:55,627] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt 4: [2023-03-17 11:58:55,627] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:58:55,627] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 11:58:55,627] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:58:55,627] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt 4: [2023-03-17 11:58:55,627] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt 4: [2023-03-17 11:58:55,627] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 11:58:55,627] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 7: [2023-03-17 11:58:55,629] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:58:55,629] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt 7: [2023-03-17 11:58:55,629] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 7: [2023-03-17 11:58:55,629] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:58:55,629] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt 7: [2023-03-17 11:58:55,629] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 7: [2023-03-17 11:58:55,629] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:58:55,629] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt 7: [2023-03-17 11:58:55,630] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 7: [2023-03-17 11:58:55,630] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:58:55,630] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt 7: [2023-03-17 11:58:55,630] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 5: [2023-03-17 11:58:55,631] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:58:55,631] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:58:55,631] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:58:55,631] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:58:55,631] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt 5: [2023-03-17 11:58:55,631] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt 5: [2023-03-17 11:58:55,631] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt 5: [2023-03-17 11:58:55,631] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt 5: [2023-03-17 11:58:55,631] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 5: [2023-03-17 11:58:55,632] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 5: [2023-03-17 11:58:55,632] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 5: [2023-03-17 11:58:55,632] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 1: [2023-03-17 11:58:55,633] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:58:55,633] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:58:55,633] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:58:55,633] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:58:55,633] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:58:55,633] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:58:55,633] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt 1: [2023-03-17 11:58:55,633] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:58:55,633] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt 1: [2023-03-17 11:58:55,633] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt 1: [2023-03-17 11:58:55,633] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt 1: [2023-03-17 11:58:55,633] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 1: [2023-03-17 11:58:55,633] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt 1: [2023-03-17 11:58:55,633] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 1: [2023-03-17 11:58:55,633] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt 1: [2023-03-17 11:58:55,633] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 1: [2023-03-17 11:58:55,633] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 1: [2023-03-17 11:58:55,633] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt 1: [2023-03-17 11:58:55,633] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 1: [2023-03-17 11:58:55,633] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 1: [2023-03-17 11:58:55,633] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 7: [2023-03-17 11:58:55,635] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:58:55,635] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:58:55,635] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt 7: [2023-03-17 11:58:55,635] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt 7: [2023-03-17 11:58:55,635] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 7: [2023-03-17 11:58:55,635] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 7: [2023-03-17 11:58:55,635] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:58:55,635] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:58:55,635] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt 7: [2023-03-17 11:58:55,635] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt 7: [2023-03-17 11:58:55,635] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 7: [2023-03-17 11:58:55,635] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 1: [2023-03-17 11:58:55,635] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:58:55,635] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt 1: [2023-03-17 11:58:55,635] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: [2023-03-17 11:58:55,645] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt 0: [2023-03-17 11:58:55,645] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 6: [2023-03-17 11:58:55,646] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:58:55,646] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt 6: [2023-03-17 11:58:55,646] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 6: [2023-03-17 11:58:55,647] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:58:55,647] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt 6: [2023-03-17 11:58:55,647] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 6: [2023-03-17 11:58:55,647] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:58:55,647] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt 6: [2023-03-17 11:58:55,647] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 6: [2023-03-17 11:58:55,648] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:58:55,648] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt 6: [2023-03-17 11:58:55,648] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:58:55,648] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 6: [2023-03-17 11:58:55,648] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt 6: [2023-03-17 11:58:55,648] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:58:55,648] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 6: [2023-03-17 11:58:55,648] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt 6: [2023-03-17 11:58:55,648] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 5: [2023-03-17 11:58:55,668] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:58:55,668] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:58:55,668] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt 5: [2023-03-17 11:58:55,668] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt 5: [2023-03-17 11:58:55,668] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 5: [2023-03-17 11:58:55,668] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 6: [2023-03-17 11:58:55,668] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:58:55,668] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:58:55,668] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt 6: [2023-03-17 11:58:55,668] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt 6: [2023-03-17 11:58:55,668] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 6: [2023-03-17 11:58:55,668] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 11:58:55,761] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:58:55,761] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt 4: [2023-03-17 11:58:55,761] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 11:58:55,761] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:58:55,761] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step10000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt 4: [2023-03-17 11:58:55,761] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: successfully saved checkpoint at iteration 10000 to checkpoints_146m91b100mdedup 7: time (ms) | save-checkpoint: 1089.35 7: iteration 10100/ 173500 | consumed samples: 2585600 | consumed tokens: 5295308800 | elapsed time per iteration (s): 0.39 | learning rate: 1.989E-04 | global batch size: 256 | lm loss: 3.720509E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 660.514 | TFLOPs: 30.83 | 7: iteration 10200/ 173500 | consumed samples: 2611200 | consumed tokens: 5347737600 | elapsed time per iteration (s): 0.38 | learning rate: 1.989E-04 | global batch size: 256 | lm loss: 3.717102E+00 | grad norm: 0.284 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 679.326 | TFLOPs: 31.71 | 7: iteration 10300/ 173500 | consumed samples: 2636800 | consumed tokens: 5400166400 | elapsed time per iteration (s): 0.38 | learning rate: 1.989E-04 | global batch size: 256 | lm loss: 3.712398E+00 | grad norm: 0.264 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 676.481 | TFLOPs: 31.58 | 7: iteration 10400/ 173500 | consumed samples: 2662400 | consumed tokens: 5452595200 | elapsed time per iteration (s): 0.38 | learning rate: 1.989E-04 | global batch size: 256 | lm loss: 3.711221E+00 | grad norm: 0.330 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 667.735 | TFLOPs: 31.17 | 7: iteration 10500/ 173500 | consumed samples: 2688000 | consumed tokens: 5505024000 | elapsed time per iteration (s): 0.38 | learning rate: 1.988E-04 | global batch size: 256 | lm loss: 3.707015E+00 | grad norm: 0.271 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 670.716 | TFLOPs: 31.31 | 7: iteration 10600/ 173500 | consumed samples: 2713600 | consumed tokens: 5557452800 | elapsed time per iteration (s): 0.38 | learning rate: 1.988E-04 | global batch size: 256 | lm loss: 3.704286E+00 | grad norm: 0.267 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 677.825 | TFLOPs: 31.64 | 7: iteration 10700/ 173500 | consumed samples: 2739200 | consumed tokens: 5609881600 | elapsed time per iteration (s): 0.37 | learning rate: 1.988E-04 | global batch size: 256 | lm loss: 3.698886E+00 | grad norm: 0.327 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 690.656 | TFLOPs: 32.24 | 7: iteration 10800/ 173500 | consumed samples: 2764800 | consumed tokens: 5662310400 | elapsed time per iteration (s): 0.37 | learning rate: 1.988E-04 | global batch size: 256 | lm loss: 3.699309E+00 | grad norm: 0.320 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 685.666 | TFLOPs: 32.00 | 7: iteration 10900/ 173500 | consumed samples: 2790400 | consumed tokens: 5714739200 | elapsed time per iteration (s): 0.37 | learning rate: 1.987E-04 | global batch size: 256 | lm loss: 3.693415E+00 | grad norm: 0.285 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.033 | TFLOPs: 32.02 | 7: iteration 11000/ 173500 | consumed samples: 2816000 | consumed tokens: 5767168000 | elapsed time per iteration (s): 0.37 | learning rate: 1.987E-04 | global batch size: 256 | lm loss: 3.690692E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.387 | TFLOPs: 32.08 | 7: iteration 11100/ 173500 | consumed samples: 2841600 | consumed tokens: 5819596800 | elapsed time per iteration (s): 0.37 | learning rate: 1.987E-04 | global batch size: 256 | lm loss: 3.687029E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 689.875 | TFLOPs: 32.20 | 7: iteration 11200/ 173500 | consumed samples: 2867200 | consumed tokens: 5872025600 | elapsed time per iteration (s): 0.37 | learning rate: 1.987E-04 | global batch size: 256 | lm loss: 3.685386E+00 | grad norm: 0.332 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.924 | TFLOPs: 32.06 | 7: iteration 11300/ 173500 | consumed samples: 2892800 | consumed tokens: 5924454400 | elapsed time per iteration (s): 0.37 | learning rate: 1.986E-04 | global batch size: 256 | lm loss: 3.683362E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.992 | TFLOPs: 32.07 | 7: iteration 11400/ 173500 | consumed samples: 2918400 | consumed tokens: 5976883200 | elapsed time per iteration (s): 0.37 | learning rate: 1.986E-04 | global batch size: 256 | lm loss: 3.679672E+00 | grad norm: 0.275 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.111 | TFLOPs: 32.03 | 7: iteration 11500/ 173500 | consumed samples: 2944000 | consumed tokens: 6029312000 | elapsed time per iteration (s): 0.37 | learning rate: 1.986E-04 | global batch size: 256 | lm loss: 3.676794E+00 | grad norm: 0.257 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.916 | TFLOPs: 32.11 | 7: iteration 11600/ 173500 | consumed samples: 2969600 | consumed tokens: 6081740800 | elapsed time per iteration (s): 0.37 | learning rate: 1.985E-04 | global batch size: 256 | lm loss: 3.675647E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.415 | TFLOPs: 32.13 | 7: iteration 11700/ 173500 | consumed samples: 2995200 | consumed tokens: 6134169600 | elapsed time per iteration (s): 0.37 | learning rate: 1.985E-04 | global batch size: 256 | lm loss: 3.669737E+00 | grad norm: 0.277 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.773 | TFLOPs: 32.06 | 7: iteration 11800/ 173500 | consumed samples: 3020800 | consumed tokens: 6186598400 | elapsed time per iteration (s): 0.37 | learning rate: 1.985E-04 | global batch size: 256 | lm loss: 3.668661E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.350 | TFLOPs: 32.04 | 7: iteration 11900/ 173500 | consumed samples: 3046400 | consumed tokens: 6239027200 | elapsed time per iteration (s): 0.37 | learning rate: 1.984E-04 | global batch size: 256 | lm loss: 3.668566E+00 | grad norm: 0.283 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.670 | TFLOPs: 32.05 | 0: [2023-03-17 12:11:24,550] [INFO] [logging.py:68:log_dist] [Rank 0] step=12000, skipped=0, lr=[0.0001984184547955352, 0.0001984184547955352, 0.0001984184547955352], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 12000/ 173500 | consumed samples: 3072000 | consumed tokens: 6291456000 | elapsed time per iteration (s): 0.37 | learning rate: 1.984E-04 | global batch size: 256 | lm loss: 3.661459E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.933 | TFLOPs: 32.06 | 0: steps: 12000 loss: 3.6426 iter time (s): 0.372 samples/sec: 687.673 7: iteration 12100/ 173500 | consumed samples: 3097600 | consumed tokens: 6343884800 | elapsed time per iteration (s): 0.37 | learning rate: 1.984E-04 | global batch size: 256 | lm loss: 3.661978E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.299 | TFLOPs: 32.08 | 7: iteration 12200/ 173500 | consumed samples: 3123200 | consumed tokens: 6396313600 | elapsed time per iteration (s): 0.37 | learning rate: 1.984E-04 | global batch size: 256 | lm loss: 3.657598E+00 | grad norm: 0.246 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 685.059 | TFLOPs: 31.98 | 7: iteration 12300/ 173500 | consumed samples: 3148800 | consumed tokens: 6448742400 | elapsed time per iteration (s): 0.37 | learning rate: 1.983E-04 | global batch size: 256 | lm loss: 3.654133E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.493 | TFLOPs: 32.09 | 7: iteration 12400/ 173500 | consumed samples: 3174400 | consumed tokens: 6501171200 | elapsed time per iteration (s): 0.37 | learning rate: 1.983E-04 | global batch size: 256 | lm loss: 3.652075E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.644 | TFLOPs: 32.10 | 7: iteration 12500/ 173500 | consumed samples: 3200000 | consumed tokens: 6553600000 | elapsed time per iteration (s): 0.37 | learning rate: 1.983E-04 | global batch size: 256 | lm loss: 3.650657E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.825 | TFLOPs: 32.15 | 7: iteration 12600/ 173500 | consumed samples: 3225600 | consumed tokens: 6606028800 | elapsed time per iteration (s): 0.37 | learning rate: 1.982E-04 | global batch size: 256 | lm loss: 3.650177E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.368 | TFLOPs: 32.08 | 7: iteration 12700/ 173500 | consumed samples: 3251200 | consumed tokens: 6658457600 | elapsed time per iteration (s): 0.37 | learning rate: 1.982E-04 | global batch size: 256 | lm loss: 3.645391E+00 | grad norm: 0.270 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 685.798 | TFLOPs: 32.01 | 7: iteration 12800/ 173500 | consumed samples: 3276800 | consumed tokens: 6710886400 | elapsed time per iteration (s): 0.37 | learning rate: 1.982E-04 | global batch size: 256 | lm loss: 3.645253E+00 | grad norm: 0.281 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.314 | TFLOPs: 32.08 | 7: iteration 12900/ 173500 | consumed samples: 3302400 | consumed tokens: 6763315200 | elapsed time per iteration (s): 0.37 | learning rate: 1.981E-04 | global batch size: 256 | lm loss: 3.642820E+00 | grad norm: 0.362 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 683.770 | TFLOPs: 31.92 | 7: iteration 13000/ 173500 | consumed samples: 3328000 | consumed tokens: 6815744000 | elapsed time per iteration (s): 0.37 | learning rate: 1.981E-04 | global batch size: 256 | lm loss: 3.644554E+00 | grad norm: 0.336 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.361 | TFLOPs: 32.04 | 7: iteration 13100/ 173500 | consumed samples: 3353600 | consumed tokens: 6868172800 | elapsed time per iteration (s): 0.37 | learning rate: 1.981E-04 | global batch size: 256 | lm loss: 3.637746E+00 | grad norm: 0.267 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 685.628 | TFLOPs: 32.00 | 7: iteration 13200/ 173500 | consumed samples: 3379200 | consumed tokens: 6920601600 | elapsed time per iteration (s): 0.37 | learning rate: 1.980E-04 | global batch size: 256 | lm loss: 3.632468E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.165 | TFLOPs: 32.07 | 7: iteration 13300/ 173500 | consumed samples: 3404800 | consumed tokens: 6973030400 | elapsed time per iteration (s): 0.37 | learning rate: 1.980E-04 | global batch size: 256 | lm loss: 3.630454E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.979 | TFLOPs: 32.16 | 7: iteration 13400/ 173500 | consumed samples: 3430400 | consumed tokens: 7025459200 | elapsed time per iteration (s): 0.37 | learning rate: 1.980E-04 | global batch size: 256 | lm loss: 3.630440E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.931 | TFLOPs: 32.16 | 7: iteration 13500/ 173500 | consumed samples: 3456000 | consumed tokens: 7077888000 | elapsed time per iteration (s): 0.37 | learning rate: 1.979E-04 | global batch size: 256 | lm loss: 3.626151E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 689.723 | TFLOPs: 32.19 | 7: iteration 13600/ 173500 | consumed samples: 3481600 | consumed tokens: 7130316800 | elapsed time per iteration (s): 0.37 | learning rate: 1.979E-04 | global batch size: 256 | lm loss: 3.620343E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 689.217 | TFLOPs: 32.17 | 7: iteration 13700/ 173500 | consumed samples: 3507200 | consumed tokens: 7182745600 | elapsed time per iteration (s): 0.37 | learning rate: 1.979E-04 | global batch size: 256 | lm loss: 3.622935E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.891 | TFLOPs: 32.15 | 7: iteration 13800/ 173500 | consumed samples: 3532800 | consumed tokens: 7235174400 | elapsed time per iteration (s): 0.37 | learning rate: 1.978E-04 | global batch size: 256 | lm loss: 3.621824E+00 | grad norm: 0.269 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.292 | TFLOPs: 32.13 | 7: iteration 13900/ 173500 | consumed samples: 3558400 | consumed tokens: 7287603200 | elapsed time per iteration (s): 0.37 | learning rate: 1.978E-04 | global batch size: 256 | lm loss: 3.617840E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.731 | TFLOPs: 32.15 | 0: [2023-03-17 12:23:49,063] [INFO] [logging.py:68:log_dist] [Rank 0] step=14000, skipped=0, lr=[0.00019774496681175836, 0.00019774496681175836, 0.00019774496681175836], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 14000/ 173500 | consumed samples: 3584000 | consumed tokens: 7340032000 | elapsed time per iteration (s): 0.37 | learning rate: 1.977E-04 | global batch size: 256 | lm loss: 3.617469E+00 | grad norm: 0.332 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 691.575 | TFLOPs: 32.28 | 0: steps: 14000 loss: 3.6365 iter time (s): 0.370 samples/sec: 691.591 7: iteration 14100/ 173500 | consumed samples: 3609600 | consumed tokens: 7392460800 | elapsed time per iteration (s): 0.37 | learning rate: 1.977E-04 | global batch size: 256 | lm loss: 3.614543E+00 | grad norm: 0.283 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.154 | TFLOPs: 32.12 | 7: iteration 14200/ 173500 | consumed samples: 3635200 | consumed tokens: 7444889600 | elapsed time per iteration (s): 0.37 | learning rate: 1.977E-04 | global batch size: 256 | lm loss: 3.611135E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 689.882 | TFLOPs: 32.20 | 7: iteration 14300/ 173500 | consumed samples: 3660800 | consumed tokens: 7497318400 | elapsed time per iteration (s): 0.37 | learning rate: 1.976E-04 | global batch size: 256 | lm loss: 3.611133E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 690.361 | TFLOPs: 32.22 | 7: iteration 14400/ 173500 | consumed samples: 3686400 | consumed tokens: 7549747200 | elapsed time per iteration (s): 0.37 | learning rate: 1.976E-04 | global batch size: 256 | lm loss: 3.607121E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 690.304 | TFLOPs: 32.22 | 7: iteration 14500/ 173500 | consumed samples: 3712000 | consumed tokens: 7602176000 | elapsed time per iteration (s): 0.37 | learning rate: 1.976E-04 | global batch size: 256 | lm loss: 3.603716E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 689.267 | TFLOPs: 32.17 | 7: iteration 14600/ 173500 | consumed samples: 3737600 | consumed tokens: 7654604800 | elapsed time per iteration (s): 0.37 | learning rate: 1.975E-04 | global batch size: 256 | lm loss: 3.603564E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 684.604 | TFLOPs: 31.95 | 7: iteration 14700/ 173500 | consumed samples: 3763200 | consumed tokens: 7707033600 | elapsed time per iteration (s): 0.37 | learning rate: 1.975E-04 | global batch size: 256 | lm loss: 3.604368E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 690.110 | TFLOPs: 32.21 | 7: iteration 14800/ 173500 | consumed samples: 3788800 | consumed tokens: 7759462400 | elapsed time per iteration (s): 0.37 | learning rate: 1.974E-04 | global batch size: 256 | lm loss: 3.600735E+00 | grad norm: 0.284 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.898 | TFLOPs: 32.16 | 7: iteration 14900/ 173500 | consumed samples: 3814400 | consumed tokens: 7811891200 | elapsed time per iteration (s): 0.37 | learning rate: 1.974E-04 | global batch size: 256 | lm loss: 3.599626E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 691.562 | TFLOPs: 32.28 | 7: iteration 15000/ 173500 | consumed samples: 3840000 | consumed tokens: 7864320000 | elapsed time per iteration (s): 0.37 | learning rate: 1.974E-04 | global batch size: 256 | lm loss: 3.598206E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.440 | TFLOPs: 32.09 | 7: iteration 15100/ 173500 | consumed samples: 3865600 | consumed tokens: 7916748800 | elapsed time per iteration (s): 0.37 | learning rate: 1.973E-04 | global batch size: 256 | lm loss: 3.594430E+00 | grad norm: 0.327 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 689.729 | TFLOPs: 32.19 | 7: iteration 15200/ 173500 | consumed samples: 3891200 | consumed tokens: 7969177600 | elapsed time per iteration (s): 0.37 | learning rate: 1.973E-04 | global batch size: 256 | lm loss: 3.594669E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.979 | TFLOPs: 32.11 | 7: iteration 15300/ 173500 | consumed samples: 3916800 | consumed tokens: 8021606400 | elapsed time per iteration (s): 0.37 | learning rate: 1.972E-04 | global batch size: 256 | lm loss: 3.590747E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.961 | TFLOPs: 32.16 | 7: iteration 15400/ 173500 | consumed samples: 3942400 | consumed tokens: 8074035200 | elapsed time per iteration (s): 0.37 | learning rate: 1.972E-04 | global batch size: 256 | lm loss: 3.587725E+00 | grad norm: 0.277 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 690.940 | TFLOPs: 32.25 | 7: iteration 15500/ 173500 | consumed samples: 3968000 | consumed tokens: 8126464000 | elapsed time per iteration (s): 0.37 | learning rate: 1.972E-04 | global batch size: 256 | lm loss: 3.586178E+00 | grad norm: 0.274 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 689.825 | TFLOPs: 32.20 | 7: iteration 15600/ 173500 | consumed samples: 3993600 | consumed tokens: 8178892800 | elapsed time per iteration (s): 0.37 | learning rate: 1.971E-04 | global batch size: 256 | lm loss: 3.583864E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.964 | TFLOPs: 32.16 | 7: iteration 15700/ 173500 | consumed samples: 4019200 | consumed tokens: 8231321600 | elapsed time per iteration (s): 0.37 | learning rate: 1.971E-04 | global batch size: 256 | lm loss: 3.582081E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 691.252 | TFLOPs: 32.27 | 7: iteration 15800/ 173500 | consumed samples: 4044800 | consumed tokens: 8283750400 | elapsed time per iteration (s): 0.37 | learning rate: 1.970E-04 | global batch size: 256 | lm loss: 3.582952E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 689.670 | TFLOPs: 32.19 | 7: iteration 15900/ 173500 | consumed samples: 4070400 | consumed tokens: 8336179200 | elapsed time per iteration (s): 0.37 | learning rate: 1.970E-04 | global batch size: 256 | lm loss: 3.581144E+00 | grad norm: 0.265 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 691.933 | TFLOPs: 32.30 | 0: [2023-03-17 12:36:11,719] [INFO] [logging.py:68:log_dist] [Rank 0] step=16000, skipped=0, lr=[0.00019695408064628468, 0.00019695408064628468, 0.00019695408064628468], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 16000/ 173500 | consumed samples: 4096000 | consumed tokens: 8388608000 | elapsed time per iteration (s): 0.37 | learning rate: 1.970E-04 | global batch size: 256 | lm loss: 3.578466E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.569 | TFLOPs: 32.14 | 0: steps: 16000 loss: 3.5705 iter time (s): 0.369 samples/sec: 693.242 7: iteration 16100/ 173500 | consumed samples: 4121600 | consumed tokens: 8441036800 | elapsed time per iteration (s): 0.37 | learning rate: 1.969E-04 | global batch size: 256 | lm loss: 3.575958E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 687.068 | TFLOPs: 32.07 | 7: iteration 16200/ 173500 | consumed samples: 4147200 | consumed tokens: 8493465600 | elapsed time per iteration (s): 0.37 | learning rate: 1.969E-04 | global batch size: 256 | lm loss: 3.577761E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 691.414 | TFLOPs: 32.27 | 7: iteration 16300/ 173500 | consumed samples: 4172800 | consumed tokens: 8545894400 | elapsed time per iteration (s): 0.37 | learning rate: 1.968E-04 | global batch size: 256 | lm loss: 3.574610E+00 | grad norm: 0.276 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 689.181 | TFLOPs: 32.17 | 7: iteration 16400/ 173500 | consumed samples: 4198400 | consumed tokens: 8598323200 | elapsed time per iteration (s): 0.37 | learning rate: 1.968E-04 | global batch size: 256 | lm loss: 3.572135E+00 | grad norm: 0.286 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 690.955 | TFLOPs: 32.25 | 7: iteration 16500/ 173500 | consumed samples: 4224000 | consumed tokens: 8650752000 | elapsed time per iteration (s): 0.37 | learning rate: 1.967E-04 | global batch size: 256 | lm loss: 3.570898E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 691.273 | TFLOPs: 32.27 | 7: iteration 16600/ 173500 | consumed samples: 4249600 | consumed tokens: 8703180800 | elapsed time per iteration (s): 0.37 | learning rate: 1.967E-04 | global batch size: 256 | lm loss: 3.566239E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 692.291 | TFLOPs: 32.31 | 7: iteration 16700/ 173500 | consumed samples: 4275200 | consumed tokens: 8755609600 | elapsed time per iteration (s): 0.37 | learning rate: 1.966E-04 | global batch size: 256 | lm loss: 3.566593E+00 | grad norm: 0.277 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 691.279 | TFLOPs: 32.27 | 7: iteration 16800/ 173500 | consumed samples: 4300800 | consumed tokens: 8808038400 | elapsed time per iteration (s): 0.37 | learning rate: 1.966E-04 | global batch size: 256 | lm loss: 3.563437E+00 | grad norm: 0.266 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.357 | TFLOPs: 32.13 | 7: iteration 16900/ 173500 | consumed samples: 4326400 | consumed tokens: 8860467200 | elapsed time per iteration (s): 0.37 | learning rate: 1.966E-04 | global batch size: 256 | lm loss: 3.565608E+00 | grad norm: 0.278 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 690.283 | TFLOPs: 32.22 | 7: iteration 17000/ 173500 | consumed samples: 4352000 | consumed tokens: 8912896000 | elapsed time per iteration (s): 0.37 | learning rate: 1.965E-04 | global batch size: 256 | lm loss: 3.562013E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 692.819 | TFLOPs: 32.34 | 7: iteration 17100/ 173500 | consumed samples: 4377600 | consumed tokens: 8965324800 | elapsed time per iteration (s): 0.37 | learning rate: 1.965E-04 | global batch size: 256 | lm loss: 3.559929E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 691.649 | TFLOPs: 32.28 | 7: iteration 17200/ 173500 | consumed samples: 4403200 | consumed tokens: 9017753600 | elapsed time per iteration (s): 0.37 | learning rate: 1.964E-04 | global batch size: 256 | lm loss: 3.558390E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 691.979 | TFLOPs: 32.30 | 7: iteration 17300/ 173500 | consumed samples: 4428800 | consumed tokens: 9070182400 | elapsed time per iteration (s): 0.37 | learning rate: 1.964E-04 | global batch size: 256 | lm loss: 3.558090E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 691.411 | TFLOPs: 32.27 | 7: iteration 17400/ 173500 | consumed samples: 4454400 | consumed tokens: 9122611200 | elapsed time per iteration (s): 0.37 | learning rate: 1.963E-04 | global batch size: 256 | lm loss: 3.555305E+00 | grad norm: 0.279 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.157 | TFLOPs: 32.12 | 7: iteration 17500/ 173500 | consumed samples: 4480000 | consumed tokens: 9175040000 | elapsed time per iteration (s): 0.38 | learning rate: 1.963E-04 | global batch size: 256 | lm loss: 3.553943E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 681.472 | TFLOPs: 31.81 | 7: iteration 17600/ 173500 | consumed samples: 4505600 | consumed tokens: 9227468800 | elapsed time per iteration (s): 0.37 | learning rate: 1.962E-04 | global batch size: 256 | lm loss: 3.551423E+00 | grad norm: 0.272 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 691.565 | TFLOPs: 32.28 | 7: iteration 17700/ 173500 | consumed samples: 4531200 | consumed tokens: 9279897600 | elapsed time per iteration (s): 0.37 | learning rate: 1.962E-04 | global batch size: 256 | lm loss: 3.550725E+00 | grad norm: 0.342 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 693.785 | TFLOPs: 32.38 | 7: iteration 17800/ 173500 | consumed samples: 4556800 | consumed tokens: 9332326400 | elapsed time per iteration (s): 0.37 | learning rate: 1.961E-04 | global batch size: 256 | lm loss: 3.549260E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 692.173 | TFLOPs: 32.31 | 7: iteration 17900/ 173500 | consumed samples: 4582400 | consumed tokens: 9384755200 | elapsed time per iteration (s): 0.37 | learning rate: 1.961E-04 | global batch size: 256 | lm loss: 3.549862E+00 | grad norm: 0.269 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 693.105 | TFLOPs: 32.35 | 0: [2023-03-17 12:48:33,053] [INFO] [logging.py:68:log_dist] [Rank 0] step=18000, skipped=0, lr=[0.00019604685446348677, 0.00019604685446348677, 0.00019604685446348677], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 18000/ 173500 | consumed samples: 4608000 | consumed tokens: 9437184000 | elapsed time per iteration (s): 0.37 | learning rate: 1.960E-04 | global batch size: 256 | lm loss: 3.546256E+00 | grad norm: 0.262 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 692.970 | TFLOPs: 32.35 | 0: steps: 18000 loss: 3.5295 iter time (s): 0.369 samples/sec: 694.228 7: iteration 18100/ 173500 | consumed samples: 4633600 | consumed tokens: 9489612800 | elapsed time per iteration (s): 0.37 | learning rate: 1.960E-04 | global batch size: 256 | lm loss: 3.545913E+00 | grad norm: 0.272 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 692.139 | TFLOPs: 32.31 | 7: iteration 18200/ 173500 | consumed samples: 4659200 | consumed tokens: 9542041600 | elapsed time per iteration (s): 0.37 | learning rate: 1.959E-04 | global batch size: 256 | lm loss: 3.544520E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 693.361 | TFLOPs: 32.36 | 7: iteration 18300/ 173500 | consumed samples: 4684800 | consumed tokens: 9594470400 | elapsed time per iteration (s): 0.37 | learning rate: 1.959E-04 | global batch size: 256 | lm loss: 3.541939E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 691.705 | TFLOPs: 32.29 | 7: iteration 18400/ 173500 | consumed samples: 4710400 | consumed tokens: 9646899200 | elapsed time per iteration (s): 0.37 | learning rate: 1.959E-04 | global batch size: 256 | lm loss: 3.542021E+00 | grad norm: 0.278 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 686.276 | TFLOPs: 32.03 | 7: iteration 18500/ 173500 | consumed samples: 4736000 | consumed tokens: 9699328000 | elapsed time per iteration (s): 0.37 | learning rate: 1.958E-04 | global batch size: 256 | lm loss: 3.539247E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 693.180 | TFLOPs: 32.36 | 7: iteration 18600/ 173500 | consumed samples: 4761600 | consumed tokens: 9751756800 | elapsed time per iteration (s): 0.37 | learning rate: 1.958E-04 | global batch size: 256 | lm loss: 3.540379E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 691.914 | TFLOPs: 32.30 | 7: iteration 18700/ 173500 | consumed samples: 4787200 | consumed tokens: 9804185600 | elapsed time per iteration (s): 0.37 | learning rate: 1.957E-04 | global batch size: 256 | lm loss: 3.535776E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 692.646 | TFLOPs: 32.33 | 7: iteration 18800/ 173500 | consumed samples: 4812800 | consumed tokens: 9856614400 | elapsed time per iteration (s): 0.37 | learning rate: 1.957E-04 | global batch size: 256 | lm loss: 3.535850E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 692.060 | TFLOPs: 32.30 | 7: iteration 18900/ 173500 | consumed samples: 4838400 | consumed tokens: 9909043200 | elapsed time per iteration (s): 0.37 | learning rate: 1.956E-04 | global batch size: 256 | lm loss: 3.532732E+00 | grad norm: 0.279 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 693.314 | TFLOPs: 32.36 | 7: iteration 19000/ 173500 | consumed samples: 4864000 | consumed tokens: 9961472000 | elapsed time per iteration (s): 0.37 | learning rate: 1.955E-04 | global batch size: 256 | lm loss: 3.534751E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 692.279 | TFLOPs: 32.31 | 7: iteration 19100/ 173500 | consumed samples: 4889600 | consumed tokens: 10013900800 | elapsed time per iteration (s): 0.37 | learning rate: 1.955E-04 | global batch size: 256 | lm loss: 3.531019E+00 | grad norm: 0.268 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 691.097 | TFLOPs: 32.26 | 7: iteration 19200/ 173500 | consumed samples: 4915200 | consumed tokens: 10066329600 | elapsed time per iteration (s): 0.37 | learning rate: 1.954E-04 | global batch size: 256 | lm loss: 3.532661E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 692.910 | TFLOPs: 32.34 | 7: iteration 19300/ 173500 | consumed samples: 4940800 | consumed tokens: 10118758400 | elapsed time per iteration (s): 0.37 | learning rate: 1.954E-04 | global batch size: 256 | lm loss: 3.531518E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 694.177 | TFLOPs: 32.40 | 7: iteration 19400/ 173500 | consumed samples: 4966400 | consumed tokens: 10171187200 | elapsed time per iteration (s): 0.37 | learning rate: 1.953E-04 | global batch size: 256 | lm loss: 3.527509E+00 | grad norm: 0.285 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 694.128 | TFLOPs: 32.40 | 7: iteration 19500/ 173500 | consumed samples: 4992000 | consumed tokens: 10223616000 | elapsed time per iteration (s): 0.37 | learning rate: 1.953E-04 | global batch size: 256 | lm loss: 3.528115E+00 | grad norm: 0.280 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 693.180 | TFLOPs: 32.36 | 7: iteration 19600/ 173500 | consumed samples: 5017600 | consumed tokens: 10276044800 | elapsed time per iteration (s): 0.37 | learning rate: 1.952E-04 | global batch size: 256 | lm loss: 3.527179E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 690.570 | TFLOPs: 32.23 | 7: iteration 19700/ 173500 | consumed samples: 5043200 | consumed tokens: 10328473600 | elapsed time per iteration (s): 0.37 | learning rate: 1.952E-04 | global batch size: 256 | lm loss: 3.528934E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 692.971 | TFLOPs: 32.35 | 7: iteration 19800/ 173500 | consumed samples: 5068800 | consumed tokens: 10380902400 | elapsed time per iteration (s): 0.37 | learning rate: 1.951E-04 | global batch size: 256 | lm loss: 3.525218E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 688.976 | TFLOPs: 32.16 | 7: iteration 19900/ 173500 | consumed samples: 5094400 | consumed tokens: 10433331200 | elapsed time per iteration (s): 0.37 | learning rate: 1.951E-04 | global batch size: 256 | lm loss: 3.521953E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 692.882 | TFLOPs: 32.34 | 0: [2023-03-17 13:00:52,783] [INFO] [logging.py:68:log_dist] [Rank 0] step=20000, skipped=0, lr=[0.00019502450208460265, 0.00019502450208460265, 0.00019502450208460265], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 20000/ 173500 | consumed samples: 5120000 | consumed tokens: 10485760000 | elapsed time per iteration (s): 0.37 | learning rate: 1.950E-04 | global batch size: 256 | lm loss: 3.522969E+00 | grad norm: 0.270 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 693.218 | TFLOPs: 32.36 | 0: steps: 20000 loss: 3.5222 iter time (s): 0.368 samples/sec: 695.879 7: ------------------------------------------------------------------------------------------------ 7: validation loss at iteration 20000 | lm loss value: 3.891626E+00 | lm loss PPL: 4.899050E+01 | 7: ------------------------------------------------------------------------------------------------ 0: saving checkpoint at iteration 20000 to checkpoints_146m91b100mdedup 0: [2023-03-17 13:00:52,956] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step20000 is begin to save! 0: [2023-03-17 13:03:56,009] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_01-model_00-model_states.pt... 0: [2023-03-17 13:03:56,190] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_01-model_00-model_states.pt. 0: [2023-03-17 13:03:56,191] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_03-model_00-model_states.pt... 0: [2023-03-17 13:03:56,207] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_03-model_00-model_states.pt. 0: [2023-03-17 13:03:56,207] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_04-model_00-model_states.pt... 0: [2023-03-17 13:03:56,224] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_04-model_00-model_states.pt. 0: [2023-03-17 13:03:56,224] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_05-model_00-model_states.pt... 0: [2023-03-17 13:03:56,240] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_05-model_00-model_states.pt. 0: [2023-03-17 13:03:56,240] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_06-model_00-model_states.pt... 0: [2023-03-17 13:03:56,256] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_06-model_00-model_states.pt. 0: [2023-03-17 13:03:56,256] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_07-model_00-model_states.pt... 0: [2023-03-17 13:03:56,272] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_07-model_00-model_states.pt. 0: [2023-03-17 13:03:56,272] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_08-model_00-model_states.pt... 0: [2023-03-17 13:03:56,288] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_08-model_00-model_states.pt. 0: [2023-03-17 13:03:56,288] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_09-model_00-model_states.pt... 0: [2023-03-17 13:03:56,304] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_09-model_00-model_states.pt. 0: [2023-03-17 13:03:56,304] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_10-model_00-model_states.pt... 0: [2023-03-17 13:03:56,320] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_10-model_00-model_states.pt. 0: [2023-03-17 13:03:56,320] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_11-model_00-model_states.pt... 0: [2023-03-17 13:03:56,336] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_11-model_00-model_states.pt. 0: [2023-03-17 13:03:56,336] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_12-model_00-model_states.pt... 0: [2023-03-17 13:03:56,352] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_12-model_00-model_states.pt. 0: [2023-03-17 13:03:56,352] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_13-model_00-model_states.pt... 0: [2023-03-17 13:03:56,368] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_13-model_00-model_states.pt. 0: [2023-03-17 13:03:56,368] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_14-model_00-model_states.pt... 0: [2023-03-17 13:03:56,384] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_14-model_00-model_states.pt. 0: [2023-03-17 13:03:56,384] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_15-model_00-model_states.pt... 0: [2023-03-17 13:03:56,400] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_15-model_00-model_states.pt. 0: [2023-03-17 13:03:56,400] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_16-model_00-model_states.pt... 0: [2023-03-17 13:03:56,416] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_16-model_00-model_states.pt. 0: [2023-03-17 13:03:56,416] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_17-model_00-model_states.pt... 0: [2023-03-17 13:03:56,431] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_17-model_00-model_states.pt. 0: [2023-03-17 13:03:56,432] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/layer_19-model_00-model_states.pt... 0: [2023-03-17 13:03:56,433] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/layer_19-model_00-model_states.pt. 0: [2023-03-17 13:03:56,434] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_146m91b100mdedup/global_step20000/mp_rank_00_model_states.pt 0: [2023-03-17 13:03:56,434] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/mp_rank_00_model_states.pt... 0: [2023-03-17 13:03:56,442] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/mp_rank_00_model_states.pt. 0: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt... 0: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt... 0: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt... 0: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt... 0: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt... 0: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt... 0: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt... 0: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt... 1: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt... 1: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt... 1: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt... 1: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt... 1: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt... 3: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt... 3: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt... 3: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt... 3: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt... 3: [2023-03-17 13:03:56,483] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt... 4: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt... 4: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt... 4: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt... 4: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt... 4: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt... 5: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt... 5: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt... 5: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt... 5: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt... 5: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt... 6: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt... 6: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt... 6: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt... 6: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt... 6: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt... 7: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt... 7: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt... 7: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt... 7: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt... 7: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt... 2: [2023-03-17 13:03:56,483] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt... 2: [2023-03-17 13:03:56,483] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt... 2: [2023-03-17 13:03:56,483] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt... 2: [2023-03-17 13:03:56,483] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt... 2: [2023-03-17 13:03:56,483] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt... 1: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt... 1: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt... 3: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt... 3: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt... 4: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt... 4: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt... 4: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt... 5: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt... 5: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt... 5: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt... 6: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt... 6: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt... 7: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt... 7: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt... 7: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt... 2: [2023-03-17 13:03:56,483] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt... 2: [2023-03-17 13:03:56,483] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt... 2: [2023-03-17 13:03:56,483] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt... 1: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt... 3: [2023-03-17 13:03:56,483] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt... 6: [2023-03-17 13:03:56,482] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt... 0: [2023-03-17 13:03:56,517] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt. 0: [2023-03-17 13:03:56,517] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt. 0: [2023-03-17 13:03:56,517] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt 0: [2023-03-17 13:03:56,517] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: [2023-03-17 13:03:56,518] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt. 0: [2023-03-17 13:03:56,518] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt 0: [2023-03-17 13:03:56,518] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: [2023-03-17 13:03:56,520] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt. 0: [2023-03-17 13:03:56,520] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt 0: [2023-03-17 13:03:56,520] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: [2023-03-17 13:03:56,520] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt. 0: [2023-03-17 13:03:56,521] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt 0: [2023-03-17 13:03:56,521] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: [2023-03-17 13:03:56,521] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt. 0: [2023-03-17 13:03:56,521] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt 0: [2023-03-17 13:03:56,521] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: [2023-03-17 13:03:56,521] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt. 0: [2023-03-17 13:03:56,521] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt 0: [2023-03-17 13:03:56,521] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: [2023-03-17 13:03:56,522] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt. 0: [2023-03-17 13:03:56,522] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt 0: [2023-03-17 13:03:56,522] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: [2023-03-17 13:03:56,555] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt 0: [2023-03-17 13:03:56,555] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 3: [2023-03-17 13:03:56,577] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt. 3: [2023-03-17 13:03:56,577] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt. 3: [2023-03-17 13:03:56,577] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt. 3: [2023-03-17 13:03:56,577] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt. 3: [2023-03-17 13:03:56,577] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt. 3: [2023-03-17 13:03:56,577] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt. 3: [2023-03-17 13:03:56,577] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt. 3: [2023-03-17 13:03:56,578] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt 3: [2023-03-17 13:03:56,578] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt 3: [2023-03-17 13:03:56,578] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt 3: [2023-03-17 13:03:56,578] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt 3: [2023-03-17 13:03:56,578] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt 3: [2023-03-17 13:03:56,578] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt 3: [2023-03-17 13:03:56,578] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt 3: [2023-03-17 13:03:56,578] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 3: [2023-03-17 13:03:56,578] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 3: [2023-03-17 13:03:56,578] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 3: [2023-03-17 13:03:56,578] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 3: [2023-03-17 13:03:56,578] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 3: [2023-03-17 13:03:56,578] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 3: [2023-03-17 13:03:56,578] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 2: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt. 2: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt. 2: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt. 2: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt. 2: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt. 2: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt. 2: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt. 2: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt. 2: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt 2: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt 2: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt 2: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt 2: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 2: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt 2: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt 2: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt 2: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt 2: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 2: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 2: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 2: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 2: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 2: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 2: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 4: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt. 4: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt. 4: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt. 4: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt. 4: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt. 4: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt. 4: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt. 4: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt. 4: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt 4: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt 4: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt 4: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt 4: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt 4: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt 4: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt 4: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt 4: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 4: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 4: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 4: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 4: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 4: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 4: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 1: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt. 4: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 1: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt 1: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt 1: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt 1: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt 1: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt 1: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt 1: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt 1: [2023-03-17 13:03:56,594] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt 1: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 1: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt. 5: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt. 5: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt. 5: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt. 5: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt. 5: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt. 5: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 1: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 1: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 1: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 1: [2023-03-17 13:03:56,594] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 13:03:56,595] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt 5: [2023-03-17 13:03:56,595] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt 5: [2023-03-17 13:03:56,595] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt 5: [2023-03-17 13:03:56,595] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt 5: [2023-03-17 13:03:56,595] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt 5: [2023-03-17 13:03:56,595] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt 5: [2023-03-17 13:03:56,595] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 13:03:56,595] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt 5: [2023-03-17 13:03:56,595] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 13:03:56,595] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt 5: [2023-03-17 13:03:56,595] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 13:03:56,595] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 13:03:56,595] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 13:03:56,595] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 13:03:56,595] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 13:03:56,595] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 6: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt. 7: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt. 7: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt. 7: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt. 7: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt. 7: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:03:56,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt 6: [2023-03-17 13:03:56,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt 6: [2023-03-17 13:03:56,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt 6: [2023-03-17 13:03:56,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt 6: [2023-03-17 13:03:56,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt 7: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt. 7: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt. 7: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:03:56,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt 7: [2023-03-17 13:03:56,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt 7: [2023-03-17 13:03:56,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt 7: [2023-03-17 13:03:56,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt 7: [2023-03-17 13:03:56,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt 7: [2023-03-17 13:03:56,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt 6: [2023-03-17 13:03:56,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt 6: [2023-03-17 13:03:56,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt 6: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 13:03:56,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt 7: [2023-03-17 13:03:56,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt 7: [2023-03-17 13:03:56,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt 6: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 6: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 6: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 6: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 6: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 6: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 6: [2023-03-17 13:03:56,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 3: [2023-03-17 13:03:57,009] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt. 3: [2023-03-17 13:03:57,009] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b100mdedup/global_step20000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt 3: [2023-03-17 13:03:57,009] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: successfully saved checkpoint at iteration 20000 to checkpoints_146m91b100mdedup 7: time (ms) | save-checkpoint: 184123.88 7: iteration 20100/ 173500 | consumed samples: 5145600 | consumed tokens: 10538188800 | elapsed time per iteration (s): 2.25 | learning rate: 1.950E-04 | global batch size: 256 | lm loss: 3.520631E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 113.544 | TFLOPs: 5.30 | 7: iteration 20200/ 173500 | consumed samples: 5171200 | consumed tokens: 10590617600 | elapsed time per iteration (s): 0.39 | learning rate: 1.949E-04 | global batch size: 256 | lm loss: 3.522366E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 661.335 | TFLOPs: 30.87 | 7: iteration 20300/ 173500 | consumed samples: 5196800 | consumed tokens: 10643046400 | elapsed time per iteration (s): 0.38 | learning rate: 1.949E-04 | global batch size: 256 | lm loss: 3.518345E+00 | grad norm: 0.283 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 667.441 | TFLOPs: 31.15 | 7: iteration 20400/ 173500 | consumed samples: 5222400 | consumed tokens: 10695475200 | elapsed time per iteration (s): 0.38 | learning rate: 1.948E-04 | global batch size: 256 | lm loss: 3.518857E+00 | grad norm: 0.275 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 674.994 | TFLOPs: 31.51 | 7: iteration 20500/ 173500 | consumed samples: 5248000 | consumed tokens: 10747904000 | elapsed time per iteration (s): 0.38 | learning rate: 1.948E-04 | global batch size: 256 | lm loss: 3.516793E+00 | grad norm: 0.335 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 677.342 | TFLOPs: 31.62 | 7: iteration 20600/ 173500 | consumed samples: 5273600 | consumed tokens: 10800332800 | elapsed time per iteration (s): 0.38 | learning rate: 1.947E-04 | global batch size: 256 | lm loss: 3.517995E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 675.758 | TFLOPs: 31.54 | 7: iteration 20700/ 173500 | consumed samples: 5299200 | consumed tokens: 10852761600 | elapsed time per iteration (s): 0.38 | learning rate: 1.946E-04 | global batch size: 256 | lm loss: 3.511813E+00 | grad norm: 0.274 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 668.914 | TFLOPs: 31.22 | 7: iteration 20800/ 173500 | consumed samples: 5324800 | consumed tokens: 10905190400 | elapsed time per iteration (s): 0.38 | learning rate: 1.946E-04 | global batch size: 256 | lm loss: 3.513307E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 674.278 | TFLOPs: 31.47 | 7: iteration 20900/ 173500 | consumed samples: 5350400 | consumed tokens: 10957619200 | elapsed time per iteration (s): 0.38 | learning rate: 1.945E-04 | global batch size: 256 | lm loss: 3.513456E+00 | grad norm: 0.263 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 677.168 | TFLOPs: 31.61 |