Model parameters: d_model 1024 ffw_size 4096 kv_size 64 n_heads 16 n_layers 18
Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 18 --hidden-size 1024 --num-attention-heads 16 --kv-channels 64 --ffn-hidden-size 4096 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 4 --global-batch-size 256 --train-samples 2_884_878 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --clip-grad 1.0 --kill-switch-path kill-switch-280m5b9400m --bf16 --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 2_884_878 --lr-warmup-samples 28_849 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_280m5b9400m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_280m5b9400m --load checkpoints_280m5b9400m --train-weighted-split-paths-path train400m.txt --valid-weighted-split-paths-path val.txt --data-impl mmap --deepspeed --deepspeed_config ds_configs/3318400.json --zero-stage 0
START 3318400: Wed 15 Mar 2023 09:57:52 PM EET
0: 
0: 
0: ======================= ROCm System Management Interface =======================
0: ================================= Concise Info =================================
0: GPU  Temp   AvgPwr  SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
0: 0    54.0c  91.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
0: 1    48.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
0: 2    45.0c  84.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
0: 3    46.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
0: 4    42.0c  85.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
0: 5    47.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
0: 6    41.0c  86.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
0: 7    48.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
0: ================================================================================
0: ============================= End of ROCm SMI Log ==============================
3: 
3: 
3: ======================= ROCm System Management Interface =======================
3: ================================= Concise Info =================================
3: GPU  Temp   AvgPwr  SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
3: 0    44.0c  92.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
3: 1    47.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
3: 2    41.0c  90.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
3: 3    47.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
3: 4    45.0c  92.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
3: 5    44.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
3: 6    42.0c  87.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
3: 7    48.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
3: ================================================================================
3: ============================= End of ROCm SMI Log ==============================
2: 
2: 
2: ======================= ROCm System Management Interface =======================
2: ================================= Concise Info =================================
2: GPU  Temp   AvgPwr  SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
2: 0    47.0c  88.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
2: 1    49.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
2: 2    36.0c  88.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
2: 3    48.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
2: 4    40.0c  98.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
2: 5    46.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
2: 6    40.0c  91.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
2: 7    41.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
2: ================================================================================
2: ============================= End of ROCm SMI Log ==============================
4: 
4: 
4: ======================= ROCm System Management Interface =======================
4: ================================= Concise Info =================================
4: GPU  Temp   AvgPwr  SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
4: 0    45.0c  84.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
4: 1    48.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
4: 2    42.0c  85.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
4: 3    46.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
4: 4    46.0c  87.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
4: 5    44.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
4: 6    44.0c  88.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
4: 7    46.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
4: ================================================================================
4: ============================= End of ROCm SMI Log ==============================
5: 
5: 
5: ======================= ROCm System Management Interface =======================
5: ================================= Concise Info =================================
5: GPU  Temp   AvgPwr  SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
5: 0    45.0c  91.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
5: 1    48.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
5: 2    47.0c  85.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
5: 3    45.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
5: 4    44.0c  95.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
5: 5    47.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
5: 6    43.0c  89.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
5: 7    44.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
5: ================================================================================
5: ============================= End of ROCm SMI Log ==============================
6: 
6: 
6: ======================= ROCm System Management Interface =======================
6: ================================= Concise Info =================================
6: GPU  Temp   AvgPwr  SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
6: 0    44.0c  89.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
6: 1    45.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
6: 2    46.0c  91.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
6: 3    52.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
6: 4    50.0c  92.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
6: 5    44.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
6: 6    46.0c  83.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
6: 7    48.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
6: ================================================================================
6: ============================= End of ROCm SMI Log ==============================
7: 
7: 
7: ======================= ROCm System Management Interface =======================
7: ================================= Concise Info =================================
7: GPU  Temp   AvgPwr  SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
7: 0    43.0c  92.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
7: 1    48.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
7: 2    42.0c  93.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
7: 3    49.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
7: 4    43.0c  88.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
7: 5    45.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
7: 6    44.0c  89.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
7: 7    48.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
7: ================================================================================
7: ============================= End of ROCm SMI Log ==============================
1: 
1: 
1: ======================= ROCm System Management Interface =======================
1: ================================= Concise Info =================================
1: GPU  Temp   AvgPwr  SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
1: 0    50.0c  83.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
1: 1    48.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
1: 2    44.0c  87.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
1: 3    40.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
1: 4    45.0c  85.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
1: 5    52.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
1: 6    39.0c  89.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
1: 7    43.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
1: ================================================================================
1: ============================= End of ROCm SMI Log ==============================
7: Launching on nid006716 (7/8), master nid006709 port 9999, GPUs 8, CUDA: True
4: Launching on nid006713 (4/8), master nid006709 port 9999, GPUs 8, CUDA: True
6: Launching on nid006715 (6/8), master nid006709 port 9999, GPUs 8, CUDA: True
3: Launching on nid006712 (3/8), master nid006709 port 9999, GPUs 8, CUDA: True
0: Launching on nid006709 (0/8), master nid006709 port 9999, GPUs 8, CUDA: True
2: Launching on nid006711 (2/8), master nid006709 port 9999, GPUs 8, CUDA: True
1: Launching on nid006710 (1/8), master nid006709 port 9999, GPUs 8, CUDA: True
5: Launching on nid006714 (5/8), master nid006709 port 9999, GPUs 8, CUDA: True
7: > setting tensorboard ...
0: using world size: 64, data-parallel-size: 64, tensor-model-parallel size: 1, pipeline-model-parallel size: 1 
0: accumulate and all-reduce gradients in fp32 for bfloat16 data type.
0: using torch.bfloat16 for parameters ...
0: ------------------------ arguments ------------------------
0:   abort_on_unmet_fused_kernel_constraints ......... False
0:   accumulate_allreduce_grads_in_fp32 .............. True
0:   adam_beta1 ...................................... 0.9
0:   adam_beta2 ...................................... 0.999
0:   adam_eps ........................................ 1e-08
0:   adlr_autoresume ................................. False
0:   adlr_autoresume_interval ........................ 1000
0:   apply_query_key_layer_scaling ................... True
0:   apply_residual_connection_post_layernorm ........ False
0:   attention_dropout ............................... 0.1
0:   attention_softmax_in_fp32 ....................... False
0:   bert_binary_head ................................ True
0:   bert_load ....................................... None
0:   bf16 ............................................ True
0:   bias_dropout_fusion ............................. True
0:   bias_gelu_fusion ................................ True
0:   biencoder_projection_dim ........................ 0
0:   biencoder_shared_query_context_model ............ False
0:   block_data_path ................................. None
0:   checkpoint_activations .......................... False
0:   checkpoint_in_cpu ............................... False
0:   checkpoint_num_layers ........................... 1
0:   clip_grad ....................................... 1.0
0:   codecarbon_dir .................................. None
0:   consumed_train_samples .......................... 0
0:   consumed_train_tokens ........................... 0
0:   consumed_valid_samples .......................... 0
0:   contigious_checkpointing ........................ False
0:   cpu_optimizer ................................... False
0:   cpu_torch_adam .................................. False
0:   curriculum_learning ............................. False
0:   data_impl ....................................... mmap
0:   data_parallel_size .............................. 64
0:   data_path ....................................... None
0:   dataloader_type ................................. single
0:   DDP_impl ........................................ local
0:   decoder_seq_length .............................. None
0:   deepscale ....................................... False
0:   deepscale_config ................................ None
0:   deepspeed ....................................... True
0:   deepspeed_activation_checkpointing .............. False
0:   deepspeed_config ................................ ds_configs/3318400.json
0:   deepspeed_mpi ................................... False
0:   distribute_checkpointed_activations ............. False
0:   distributed_backend ............................. nccl
0:   embed_layernorm ................................. False
0:   embedding_path .................................. None
0:   encoder_seq_length .............................. 2048
0:   eod_mask_loss ................................... False
0:   eval_interval ................................... 1000
0:   eval_iters ...................................... 1
0:   eval_only ....................................... None
0:   evidence_data_path .............................. None
0:   exit_duration_in_mins ........................... None
0:   exit_interval ................................... None
0:   ffn_hidden_size ................................. 4096
0:   finetune ........................................ False
0:   fp16 ............................................ False
0:   fp16_lm_cross_entropy ........................... False
0:   fp32_residual_connection ........................ False
0:   gigaflos_no_embeds .............................. 0
0:   global_batch_size ............................... 256
0:   glu_activation .................................. None
0:   hidden_dropout .................................. 0.1
0:   hidden_size ..................................... 1024
0:   hysteresis ...................................... 2
0:   ict_head_size ................................... None
0:   ict_load ........................................ None
0:   img_dim ......................................... 224
0:   indexer_batch_size .............................. 128
0:   indexer_log_interval ............................ 1000
0:   inference ....................................... False
0:   init_method_std ................................. 0.02
0:   init_method_xavier_uniform ...................... False
0:   initial_loss_scale .............................. 4294967296
0:   kill_switch_path ................................ kill-switch-280m5b9400m
0:   kv_channels ..................................... 64
0:   layer_norm_fusion ............................... True
0:   layernorm_epsilon ............................... 1e-05
0:   lazy_mpu_init ................................... None
0:   load ............................................ checkpoints_280m5b9400m
0:   local_rank ...................................... None
0:   log_batch_size_to_tensorboard ................... True
0:   log_interval .................................... 10
0:   log_learning_rate_to_tensorboard ................ True
0:   log_level ....................................... None
0:   log_level_replica ............................... None
0:   log_loss_scale_to_tensorboard ................... True
0:   log_num_zeros_in_grad ........................... False
0:   log_params_norm ................................. False
0:   log_path ........................................ None
0:   log_timers_to_tensorboard ....................... True
0:   log_validation_ppl_to_tensorboard ............... True
0:   loss_on_targets_only ............................ False
0:   loss_scale ...................................... None
0:   loss_scale_window ............................... 1000
0:   lr .............................................. 0.0002
0:   lr_decay_iters .................................. None
0:   lr_decay_samples ................................ 2884878
0:   lr_decay_style .................................. cosine
0:   lr_decay_tokens ................................. None
0:   lr_warmup_fraction .............................. None
0:   lr_warmup_iters ................................. 0
0:   lr_warmup_samples ............................... 28849
0:   make_vocab_size_divisible_by .................... 128
0:   mask_prob ....................................... 0.15
0:   masked_softmax_fusion ........................... True
0:   max_position_embeddings ......................... 2048
0:   mean_noise_span_length .......................... None
0:   memory_centric_tiled_linear ..................... False
0:   merge_file ...................................... gpt2/merges.txt
0:   micro_batch_size ................................ 4
0:   min_loss_scale .................................. 1.0
0:   min_lr .......................................... 2e-05
0:   mmap_warmup ..................................... False
0:   no_load_optim ................................... None
0:   no_load_rng ..................................... None
0:   no_save_optim ................................... None
0:   no_save_rng ..................................... None
0:   noise_density ................................... None
0:   num_attention_heads ............................. 16
0:   num_channels .................................... 3
0:   num_classes ..................................... 1000
0:   num_layers ...................................... 18
0:   num_layers_per_virtual_pipeline_stage ........... None
0:   num_workers ..................................... 2
0:   onnx_safe ....................................... None
0:   openai_gelu ..................................... False
0:   optimizer ....................................... adam
0:   optimizer_fusion ................................ True
0:   override_lr_scheduler ........................... False
0:   pad_vocab_size_to ............................... None
0:   params_dtype .................................... torch.bfloat16
0:   partition_activations ........................... False
0:   patch_dim ....................................... 16
0:   pipeline_model_parallel_size .................... 1
0:   position_embedding_type ......................... PositionEmbeddingType.absolute
0:   pp_partition_method ............................. None
0:   profile_backward ................................ False
0:   query_in_block_prob ............................. 0.1
0:   rampup_batch_size ............................... None
0:   rank ............................................ 0
0:   remote_device ................................... none
0:   reset_attention_mask ............................ False
0:   reset_position_ids .............................. False
0:   reset_progress .................................. None
0:   retriever_report_topk_accuracies ................ []
0:   retriever_score_scaling ......................... False
0:   retriever_seq_length ............................ 256
0:   reweight_loss_based_on_position_frequency ....... False
0:   sample_rate ..................................... 1.0
0:   save ............................................ checkpoints_280m5b9400m
0:   save_interval ................................... 1000
0:   scatter_gather_tensors_in_pipeline .............. True
0:   scattered_embeddings ............................ False
0:   seed ............................................ 1234
0:   seq_length ...................................... 2048
0:   sgd_momentum .................................... 0.9
0:   short_seq_prob .................................. 0.1
0:   skip_train_iteration_range ...................... None
0:   split ........................................... None
0:   split_transformers .............................. False
0:   sync_tp_duplicated_parameters ................... False
0:   synchronize_each_layer .......................... False
0:   tensor_model_parallel_size ...................... 1
0:   tensorboard_dir ................................. tensorboard_280m5b9400m
0:   tensorboard_log_interval ........................ 1
0:   tensorboard_queue_size .......................... 5
0:   test_weighted_split_paths ....................... None
0:   test_weighted_split_paths_path .................. None
0:   tile_factor ..................................... 1
0:   titles_data_path ................................ None
0:   tokenizer_name_or_path .......................... None
0:   tokenizer_type .................................. GPT2BPETokenizer
0:   train_iters ..................................... None
0:   train_samples ................................... 2884878
0:   train_tokens .................................... None
0:   train_weighted_split_names ...................... ['train']
0:   train_weighted_split_paths ...................... [['/scratch/project_462000119/data/c4_subsampled/gpt2tok_c4_en_400M_text_document']]
0:   train_weighted_split_paths_path ................. None
0:   train_weighted_split_splits ..................... [['0:1']]
0:   train_weighted_split_weights .................... [['1.0']]
0:   universal_checkpoint ............................ False
0:   use_bnb_optimizer ............................... False
0:   use_checkpoint_lr_scheduler ..................... False
0:   use_contiguous_buffers_in_ddp ................... True
0:   use_cpu_initialization .......................... None
0:   use_one_sent_docs ............................... False
0:   use_pin_memory .................................. False
0:   valid_num_workers ............................... 2
0:   valid_weighted_split_names ...................... ['validation']
0:   valid_weighted_split_paths ...................... [['/scratch/project_462000119/data/c4_validation/gpt2tok_c4validation_rerun_text_document']]
0:   valid_weighted_split_paths_path ................. None
0:   valid_weighted_split_splits ..................... [['0:1']]
0:   valid_weighted_split_weights .................... [['1.0']]
0:   virtual_pipeline_model_parallel_size ............ None
0:   vocab_extra_ids ................................. 0
0:   vocab_file ...................................... gpt2/vocab.json
0:   weight_decay .................................... 0.1
0:   world_size ...................................... 64
0:   zero_allgather_bucket_size ...................... 0.0
0:   zero_contigious_gradients ....................... False
0:   zero_reduce_bucket_size ......................... 0.0
0:   zero_reduce_scatter ............................. False
0:   zero_stage ...................................... 0
0: -------------------- end of arguments ---------------------
0: setting number of micro-batches to constant 1
0: > building GPT2BPETokenizer tokenizer ...
0:  > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
0: DeepSpeed general environment info:
0: torch install path ............... ['/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch']
0: torch version .................... 1.13.0+rocm5.2
0: torch cuda version ............... None
0: torch hip version ................ 5.2.21151-afdc89f8
0: nvcc version ..................... None
0: deepspeed install path ........... ['/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/deepspeed']
0: deepspeed info ................... 0.7.5, unknown, unknown
0: deepspeed wheel compiled w. ...... torch 1.13, hip 5.1
0: **** Git info for Megatron: git_hash=unknown git_branch=unknown ****
0: > initializing torch distributed ...
0: [2023-03-15 21:58:31,628] [INFO] [comm.py:633:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
0: > initializing tensor model parallel with size 1
0: > initializing pipeline model parallel with size 1
0: > setting random seeds to 1234 ...
0: > initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 3952 and data parallel seed: 1234
0: > compiling dataset index builder ...
0: make: Entering directory '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/data'
0: make: Nothing to be done for 'default'.
0: make: Leaving directory '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/data'
0: >>> done with dataset index builder. Compilation time: 0.102 seconds
0: > compiling and loading fused kernels ...
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.cpp -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.cpp [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.h [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_cuda.cu -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.hip [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.h [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.h [skipped, already hipified]
0: Total number of unsupported CUDA function calls: 0
0: 
0: 
0: Total number of replaced kernel launches: 87
0: ninja: no work to do.
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.cpp -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.cpp [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_cuda.cu -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.h [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.h [skipped, already hipified]
0: Total number of unsupported CUDA function calls: 0
0: 
0: 
0: Total number of replaced kernel launches: 63
0: ninja: no work to do.
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_cuda.cpp -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_cuda.cpp [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_cuda_kernel.cu -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_hip_kernel.hip [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.h [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.h [skipped, already hipified]
0: Total number of unsupported CUDA function calls: 0
0: 
0: 
0: Total number of replaced kernel launches: 67
0: [1/1] c++ layer_norm_hip_kernel.cuda.o layer_norm_cuda.o -shared -L/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/lib -lc10 -lc10_hip -ltorch_cpu -ltorch_hip -ltorch -ltorch_python -L/pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2.3/lib -lamdhip64 -o fused_mix_prec_layer_norm_cuda.so
0: >>> done with compiling and loading fused kernels. Compilation time: 24.091 seconds
0: time to initialize megatron (seconds): 2.569
0: [after megatron is initialized] datetime: 2023-03-15 21:58:58 
0: building GPT model ...
0: [2023-03-15 21:58:58,704] [INFO] [utils.py:827:see_memory_usage] Before Building Model
0: [2023-03-15 21:58:58,705] [INFO] [utils.py:828:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB 
0: [2023-03-15 21:58:58,705] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 31.43 GB, percent = 6.2%
0: SEED_LAYERS=False BASE_SEED=1234 SEED_FN=None
0: Using topology: {ProcessCoord(pipe=0, data=0, model=0): 0, ProcessCoord(pipe=0, data=1, model=0): 1, ProcessCoord(pipe=0, data=2, model=0): 2, ProcessCoord(pipe=0, data=3, model=0): 3, ProcessCoord(pipe=0, data=4, model=0): 4, ProcessCoord(pipe=0, data=5, model=0): 5, ProcessCoord(pipe=0, data=6, model=0): 6, ProcessCoord(pipe=0, data=7, model=0): 7, ProcessCoord(pipe=0, data=8, model=0): 8, ProcessCoord(pipe=0, data=9, model=0): 9, ProcessCoord(pipe=0, data=10, model=0): 10, ProcessCoord(pipe=0, data=11, model=0): 11, ProcessCoord(pipe=0, data=12, model=0): 12, ProcessCoord(pipe=0, data=13, model=0): 13, ProcessCoord(pipe=0, data=14, model=0): 14, ProcessCoord(pipe=0, data=15, model=0): 15, ProcessCoord(pipe=0, data=16, model=0): 16, ProcessCoord(pipe=0, data=17, model=0): 17, ProcessCoord(pipe=0, data=18, model=0): 18, ProcessCoord(pipe=0, data=19, model=0): 19, ProcessCoord(pipe=0, data=20, model=0): 20, ProcessCoord(pipe=0, data=21, model=0): 21, ProcessCoord(pipe=0, data=22, model=0): 22, ProcessCoord(pi
0: pe=0, data=23, model=0): 23, ProcessCoord(pipe=0, data=24, model=0): 24, ProcessCoord(pipe=0, data=25, model=0): 25, ProcessCoord(pipe=0, data=26, model=0): 26, ProcessCoord(pipe=0, data=27, model=0): 27, ProcessCoord(pipe=0, data=28, model=0): 28, ProcessCoord(pipe=0, data=29, model=0): 29, ProcessCoord(pipe=0, data=30, model=0): 30, ProcessCoord(pipe=0, data=31, model=0): 31, ProcessCoord(pipe=0, data=32, model=0): 32, ProcessCoord(pipe=0, data=33, model=0): 33, ProcessCoord(pipe=0, data=34, model=0): 34, ProcessCoord(pipe=0, data=35, model=0): 35, ProcessCoord(pipe=0, data=36, model=0): 36, ProcessCoord(pipe=0, data=37, model=0): 37, ProcessCoord(pipe=0, data=38, model=0): 38, ProcessCoord(pipe=0, data=39, model=0): 39, ProcessCoord(pipe=0, data=40, model=0): 40, ProcessCoord(pipe=0, data=41, model=0): 41, ProcessCoord(pipe=0, data=42, model=0): 42, ProcessCoord(pipe=0, data=43, model=0): 43, ProcessCoord(pipe=0, data=44, model=0): 44, ProcessCoord(pipe=0, data=45, model=0): 45, ProcessCoord(pipe=0, data=4
0: 6, model=0): 46, ProcessCoord(pipe=0, data=47, model=0): 47, ProcessCoord(pipe=0, data=48, model=0): 48, ProcessCoord(pipe=0, data=49, model=0): 49, ProcessCoord(pipe=0, data=50, model=0): 50, ProcessCoord(pipe=0, data=51, model=0): 51, ProcessCoord(pipe=0, data=52, model=0): 52, ProcessCoord(pipe=0, data=53, model=0): 53, ProcessCoord(pipe=0, data=54, model=0): 54, ProcessCoord(pipe=0, data=55, model=0): 55, ProcessCoord(pipe=0, data=56, model=0): 56, ProcessCoord(pipe=0, data=57, model=0): 57, ProcessCoord(pipe=0, data=58, model=0): 58, ProcessCoord(pipe=0, data=59, model=0): 59, ProcessCoord(pipe=0, data=60, model=0): 60, ProcessCoord(pipe=0, data=61, model=0): 61, ProcessCoord(pipe=0, data=62, model=0): 62, ProcessCoord(pipe=0, data=63, model=0): 63}
0: [2023-03-15 21:59:00,693] [INFO] [module.py:366:_partition_layers] Partitioning pipeline stages with method type:transformer
0: stage=0 layers=25
0:      0: _to_float16
0:      1: EmbeddingPipe
0:      2: <lambda>
0:      3: ParallelTransformerLayerPipe
0:      4: ParallelTransformerLayerPipe
0:      5: ParallelTransformerLayerPipe
0:      6: ParallelTransformerLayerPipe
0:      7: ParallelTransformerLayerPipe
0:      8: ParallelTransformerLayerPipe
0:      9: ParallelTransformerLayerPipe
0:     10: ParallelTransformerLayerPipe
0:     11: ParallelTransformerLayerPipe
0:     12: ParallelTransformerLayerPipe
0:     13: ParallelTransformerLayerPipe
0:     14: ParallelTransformerLayerPipe
0:     15: ParallelTransformerLayerPipe
0:     16: ParallelTransformerLayerPipe
0:     17: ParallelTransformerLayerPipe
0:     18: ParallelTransformerLayerPipe
0:     19: ParallelTransformerLayerPipe
0:     20: ParallelTransformerLayerPipe
0:     21: undo
0:     22: MixedFusedLayerNorm
0:     23: EmbeddingPipe
0:     24: float16_to_fp32
0:   loss: CrossEntropy
0: [2023-03-15 21:59:00,895] [INFO] [utils.py:827:see_memory_usage] After Building Model
0: [2023-03-15 21:59:00,895] [INFO] [utils.py:828:see_memory_usage] MA 0.53 GB         Max_MA 0.53 GB         CA 0.57 GB         Max_CA 1 GB 
0: [2023-03-15 21:59:00,896] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 31.45 GB, percent = 6.2%
0: setting training iterations to 11269
0: > learning rate decay style: cosine
0: DeepSpeed is enabled.
0: [2023-03-15 21:59:00,898] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed info: version=0.7.5, git-hash=unknown, git-branch=unknown
0: [2023-03-15 21:59:14,194] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
0: [2023-03-15 21:59:14,195] [INFO] [logging.py:68:log_dist] [Rank 0] Removing param_group that has no 'params' in the client Optimizer
0: [2023-03-15 21:59:14,195] [INFO] [logging.py:68:log_dist] [Rank 0] Using client Optimizer as basic optimizer
0: [2023-03-15 21:59:14,201] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam
0: [2023-03-15 21:59:14,201] [INFO] [logging.py:68:log_dist] [Rank 0] Creating BF16 optimizer
0: [2023-03-15 21:59:14,318] [INFO] [utils.py:827:see_memory_usage] begin bf16_optimizer
0: [2023-03-15 21:59:14,319] [INFO] [utils.py:828:see_memory_usage] MA 0.52 GB         Max_MA 0.53 GB         CA 0.57 GB         Max_CA 1 GB 
0: [2023-03-15 21:59:14,319] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 32.13 GB, percent = 6.4%
3: ninja: no work to do.
3: Time to load utils op: 0.17144203186035156 seconds
6: Time to load utils op: 0.21031498908996582 seconds
7: Time to load utils op: 0.20921635627746582 seconds
0: Time to load utils op: 0.11027336120605469 seconds
3: Time to load utils op: 0.0010654926300048828 seconds
0: Time to load utils op: 0.10276937484741211 seconds
0: Time to load utils op: 0.10281586647033691 seconds
0: Time to load utils op: 0.10351991653442383 seconds
0: Time to load utils op: 0.10327744483947754 seconds
0: Time to load utils op: 0.10352826118469238 seconds
0: Time to load utils op: 0.10354304313659668 seconds
0: Time to load utils op: 0.10361194610595703 seconds
3: Time to load utils op: 0.10242342948913574 seconds
3: Time to load utils op: 0.10216617584228516 seconds
3: Time to load utils op: 0.1015923023223877 seconds
3: Time to load utils op: 0.10169458389282227 seconds
3: Time to load utils op: 0.10215950012207031 seconds
3: Time to load utils op: 0.10169386863708496 seconds
3: Time to load utils op: 0.10167503356933594 seconds
6: Time to load utils op: 0.10226702690124512 seconds
6: Time to load utils op: 0.10262346267700195 seconds
6: Time to load utils op: 0.10271859169006348 seconds
6: Time to load utils op: 0.10250139236450195 seconds
6: Time to load utils op: 0.10264205932617188 seconds
6: Time to load utils op: 0.10289859771728516 seconds
6: Time to load utils op: 0.10301637649536133 seconds
7: Time to load utils op: 0.10228276252746582 seconds
1: Time to load utils op: 0.11133885383605957 seconds
1: Time to load utils op: 0.11049652099609375 seconds
1: Time to load utils op: 0.1105794906616211 secondsTime to load utils op: 0.11010456085205078 seconds
1: 
1: Time to load utils op: 0.11027312278747559 seconds
1: Time to load utils op: 0.11065459251403809 secondsTime to load utils op: 0.11060142517089844 seconds
1: 
1: Time to load utils op: 0.1106557846069336 seconds
7: Time to load utils op: 0.10219359397888184 seconds
7: Time to load utils op: 0.10232067108154297 seconds
7: Time to load utils op: 0.10240292549133301 secondsTime to load utils op: 0.10241007804870605 seconds
7: 
7: Time to load utils op: 0.1020669937133789 seconds
7: Time to load utils op: 0.10207653045654297 seconds
2: Time to load utils op: 0.11097121238708496 secondsTime to load utils op: 0.11098074913024902 seconds
2: 
2: Time to load utils op: 0.110992431640625 seconds
2: Time to load utils op: 0.11100411415100098 secondsTime to load utils op: 0.11100554466247559 secondsTime to load utils op: 0.11099910736083984 seconds
2: 
2: 
2: Time to load utils op: 0.1110084056854248 seconds
2: Time to load utils op: 0.1110086441040039 seconds
5: Time to load utils op: 0.11234760284423828 secondsTime to load utils op: 0.11210250854492188 seconds
5: 
5: Time to load utils op: 0.11231279373168945 secondsTime to load utils op: 0.11213088035583496 seconds
5: 
5: Time to load utils op: 0.11268305778503418 secondsTime to load utils op: 0.11213564872741699 secondsTime to load utils op: 0.11218738555908203 seconds
5: Time to load utils op: 0.11213803291320801 seconds
5: 
5: 
4: Time to load utils op: 0.11065554618835449 seconds
4: Time to load utils op: 0.11067652702331543 seconds
4: Time to load utils op: 0.11066770553588867 seconds
4: Time to load utils op: 0.11067700386047363 seconds
4: Time to load utils op: 0.11069846153259277 secondsTime to load utils op: 0.1107032299041748 seconds
4: 
4: Time to load utils op: 0.11071372032165527 seconds
4: Time to load utils op: 0.11070370674133301 seconds
3: Time to load utils op: 0.0003390312194824219 seconds
3: Time to load utils op: 0.00033354759216308594 seconds
3: Time to load utils op: 0.00032711029052734375 seconds
3: Time to load utils op: 0.00031113624572753906 seconds
3: Time to load utils op: 0.000377655029296875 seconds
3: Time to load utils op: 0.00036597251892089844 seconds
3: Time to load utils op: 0.0003902912139892578 seconds
6: Time to load utils op: 0.0003979206085205078 seconds
6: Time to load utils op: 0.0004973411560058594 seconds
6: Time to load utils op: 0.000415802001953125 seconds
6: Time to load utils op: 0.000415802001953125 seconds
6: Time to load utils op: 0.0005066394805908203 seconds
6: Time to load utils op: 0.0004954338073730469 seconds
6: Time to load utils op: 0.0005273818969726562 secondsTime to load utils op: 0.0005331039428710938 seconds
6: 
7: Time to load utils op: 0.00045943260192871094 seconds
7: Time to load utils op: 0.00038743019104003906 seconds
7: Time to load utils op: 0.0003757476806640625 seconds
7: Time to load utils op: 0.000385284423828125 seconds
7: Time to load utils op: 0.0003826618194580078 seconds
7: Time to load utils op: 0.00043201446533203125 seconds
7: Time to load utils op: 0.000415802001953125 seconds
7: Time to load utils op: 0.0003705024719238281 seconds
1: Time to load utils op: 0.0005168914794921875 seconds
1: Time to load utils op: 0.0009691715240478516 secondsTime to load utils op: 0.0009138584136962891 seconds
1: 
1: Time to load utils op: 0.000896453857421875 seconds
1: Time to load utils op: 0.0011131763458251953 seconds
1: Time to load utils op: 0.0010991096496582031 seconds
1: Time to load utils op: 0.0011289119720458984 seconds
1: Time to load utils op: 0.0009953975677490234 seconds
0: Time to load utils op: 0.0005435943603515625 seconds
0: Time to load utils op: 0.0004413127899169922 seconds
0: Time to load utils op: 0.0004467964172363281 seconds
0: Time to load utils op: 0.00043964385986328125 seconds
0: Time to load utils op: 0.00044989585876464844 seconds
0: Time to load utils op: 0.0004558563232421875 seconds
0: Time to load utils op: 0.00044727325439453125 seconds
5: Time to load utils op: 0.0007803440093994141 seconds
5: Time to load utils op: 0.0010323524475097656 secondsTime to load utils op: 0.0010428428649902344 seconds
5: 
5: Time to load utils op: 0.001180410385131836 seconds
5: Time to load utils op: 0.0011432170867919922 seconds
5: Time to load utils op: 0.0011034011840820312 seconds
5: Time to load utils op: 0.0011391639709472656 seconds
2: Time to load utils op: 0.0008594989776611328 seconds
5: Time to load utils op: 0.001142740249633789 seconds
2: Time to load utils op: 0.0014104843139648438 seconds
2: Time to load utils op: 0.0013108253479003906 secondsTime to load utils op: 0.0013263225555419922 seconds
2: 
2: Time to load utils op: 0.0013880729675292969 seconds
2: Time to load utils op: 0.0013074874877929688 seconds
2: Time to load utils op: 0.0013096332550048828 seconds
2: Time to load utils op: 0.0013210773468017578 seconds
4: Time to load utils op: 0.0007288455963134766 seconds
4: Time to load utils op: 0.0009312629699707031 seconds
4: Time to load utils op: 0.0011856555938720703 seconds
4: Time to load utils op: 0.001066446304321289 seconds
4: Time to load utils op: 0.0012581348419189453 seconds
4: Time to load utils op: 0.0010874271392822266 secondsTime to load utils op: 0.0011210441589355469 seconds
4: 
4: Time to load utils op: 0.001285552978515625 seconds
0: [2023-03-15 21:59:14,553] [INFO] [utils.py:827:see_memory_usage] before initializing group 0
0: [2023-03-15 21:59:14,554] [INFO] [utils.py:828:see_memory_usage] MA 0.52 GB         Max_MA 0.52 GB         CA 0.57 GB         Max_CA 1 GB 
0: [2023-03-15 21:59:14,554] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 32.26 GB, percent = 6.4%
0: [2023-03-15 21:59:14,681] [INFO] [utils.py:827:see_memory_usage] after initializing group 0
0: [2023-03-15 21:59:14,682] [INFO] [utils.py:828:see_memory_usage] MA 1.14 GB         Max_MA 1.14 GB         CA 1.48 GB         Max_CA 1 GB 
0: [2023-03-15 21:59:14,682] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 32.27 GB, percent = 6.4%
0: [2023-03-15 21:59:14,785] [INFO] [utils.py:827:see_memory_usage] before initializing group 1
0: [2023-03-15 21:59:14,786] [INFO] [utils.py:828:see_memory_usage] MA 1.14 GB         Max_MA 1.14 GB         CA 1.48 GB         Max_CA 1 GB 
0: [2023-03-15 21:59:14,786] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 32.27 GB, percent = 6.4%
0: [2023-03-15 21:59:14,889] [INFO] [utils.py:827:see_memory_usage] after initializing group 1
0: [2023-03-15 21:59:14,890] [INFO] [utils.py:828:see_memory_usage] MA 1.58 GB         Max_MA 1.58 GB         CA 2.14 GB         Max_CA 2 GB 
0: [2023-03-15 21:59:14,890] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 32.27 GB, percent = 6.4%
0: [2023-03-15 21:59:14,991] [INFO] [utils.py:827:see_memory_usage] before initializing group 2
0: [2023-03-15 21:59:14,991] [INFO] [utils.py:828:see_memory_usage] MA 1.58 GB         Max_MA 1.58 GB         CA 2.14 GB         Max_CA 2 GB 
0: [2023-03-15 21:59:14,992] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 32.27 GB, percent = 6.4%
0: [2023-03-15 21:59:15,096] [INFO] [utils.py:827:see_memory_usage] after initializing group 2
0: [2023-03-15 21:59:15,096] [INFO] [utils.py:828:see_memory_usage] MA 1.58 GB         Max_MA 1.58 GB         CA 2.14 GB         Max_CA 2 GB 
0: [2023-03-15 21:59:15,096] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 32.27 GB, percent = 6.4%
0: [2023-03-15 21:59:15,197] [INFO] [utils.py:827:see_memory_usage] before initialize_optimizer
0: [2023-03-15 21:59:15,198] [INFO] [utils.py:828:see_memory_usage] MA 1.58 GB         Max_MA 1.58 GB         CA 2.14 GB         Max_CA 2 GB 
0: [2023-03-15 21:59:15,198] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 32.27 GB, percent = 6.4%
0: [2023-03-15 21:59:15,304] [INFO] [utils.py:827:see_memory_usage] end initialize_optimizer
0: [2023-03-15 21:59:15,305] [INFO] [utils.py:828:see_memory_usage] MA 1.62 GB         Max_MA 1.62 GB         CA 2.14 GB         Max_CA 2 GB 
0: [2023-03-15 21:59:15,305] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 32.27 GB, percent = 6.4%
0: [2023-03-15 21:59:15,407] [INFO] [utils.py:827:see_memory_usage] end bf16_optimizer
0: [2023-03-15 21:59:15,408] [INFO] [utils.py:828:see_memory_usage] MA 1.62 GB         Max_MA 1.62 GB         CA 2.14 GB         Max_CA 2 GB 
0: [2023-03-15 21:59:15,408] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 32.27 GB, percent = 6.4%
0: [2023-03-15 21:59:15,408] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Final Optimizer = FusedAdam
0: [2023-03-15 21:59:15,408] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed using client LR scheduler
0: [2023-03-15 21:59:15,409] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed LR Scheduler = <megatron.learning_rates.AnnealingLR object at 0x14a938b11fd0>
0: [2023-03-15 21:59:15,409] [INFO] [logging.py:68:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0, 0.0, 0.0], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
0: [2023-03-15 21:59:15,409] [INFO] [config.py:1007:print] DeepSpeedEngine configuration:
0: [2023-03-15 21:59:15,409] [INFO] [config.py:1011:print]   activation_checkpointing_config  {
0:     "partition_activations": false, 
0:     "contiguous_memory_optimization": false, 
0:     "cpu_checkpointing": false, 
0:     "number_checkpoints": null, 
0:     "synchronize_checkpoint_boundary": false, 
0:     "profile": false
0: }
0: [2023-03-15 21:59:15,409] [INFO] [config.py:1011:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
0: [2023-03-15 21:59:15,409] [INFO] [config.py:1011:print]   amp_enabled .................. False
0: [2023-03-15 21:59:15,409] [INFO] [config.py:1011:print]   amp_params ................... False
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   autotuning_config ............ {
0:     "enabled": false, 
0:     "start_step": null, 
0:     "end_step": null, 
0:     "metric_path": null, 
0:     "arg_mappings": null, 
0:     "metric": "throughput", 
0:     "model_info": null, 
0:     "results_dir": "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/autotuning_results", 
0:     "exps_dir": "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/autotuning_exps", 
0:     "overwrite": true, 
0:     "fast": true, 
0:     "start_profile_step": 3, 
0:     "end_profile_step": 5, 
0:     "tuner_type": "gridsearch", 
0:     "tuner_early_stopping": 5, 
0:     "tuner_num_trials": 50, 
0:     "model_info_path": null, 
0:     "mp_size": 1, 
0:     "max_train_batch_size": null, 
0:     "min_train_batch_size": 1, 
0:     "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
0:     "min_train_micro_batch_size_per_gpu": 1, 
0:     "num_tuning_micro_batch_sizes": 3
0: }
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   bfloat16_enabled ............. True
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   checkpoint_parallel_write_pipeline  False
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   checkpoint_tag_validation_enabled  True
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   checkpoint_tag_validation_fail  False
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x14a7d4822070>
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   communication_data_type ...... None
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_pa
0: rameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   curriculum_enabled ........... False
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   curriculum_params ............ False
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   dataloader_drop_last ......... False
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   disable_allgather ............ False
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   dump_state ................... False
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   dynamic_loss_scale_args ...... None
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   eigenvalue_enabled ........... False
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   eigenvalue_gas_boundary_resolution  1
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   eigenvalue_layer_name ........ bert.encoder.layer
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   eigenvalue_layer_num ......... 0
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   eigenvalue_max_iter .......... 100
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   eigenvalue_stability ......... 1e-06
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   eigenvalue_tol ............... 0.01
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   eigenvalue_verbose ........... False
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   elasticity_enabled ........... False
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   flops_profiler_config ........ {
0:     "enabled": false, 
0:     "profile_step": 1, 
0:     "module_depth": -1, 
0:     "top_modules": 1, 
0:     "detailed": true, 
0:     "output_file": null
0: }
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   fp16_auto_cast ............... None
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   fp16_enabled ................. False
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   fp16_master_weights_and_gradients  False
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   global_rank .................. 0
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   gradient_accumulation_steps .. 1
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   gradient_clipping ............ 1.0
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   gradient_predivide_factor .... 1.0
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   initial_dynamic_scale ........ 1
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   load_universal_checkpoint .... False
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   loss_scale ................... 1.0
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   memory_breakdown ............. False
0: [2023-03-15 21:59:15,410] [INFO] [config.py:1011:print]   monitor_config ............... <deepspeed.monitor.config.DeepSpeedMonitorConfig object at 0x14a7d4822100>
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   nebula_config ................ {
0:     "enabled": false, 
0:     "persistent_storage_path": null, 
0:     "persistent_time_interval": 100, 
0:     "num_of_version_in_retention": 2, 
0:     "enable_nebula_load": true, 
0:     "load_path": null
0: }
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   optimizer_legacy_fusion ...... False
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   optimizer_name ............... None
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   optimizer_params ............. None
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   pld_enabled .................. False
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   pld_params ................... False
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   prescale_gradients ........... False
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   scheduler_name ............... None
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   scheduler_params ............. None
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   sparse_attention ............. None
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   sparse_gradients_enabled ..... False
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   steps_per_print .............. 2000
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   train_batch_size ............. 256
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   train_micro_batch_size_per_gpu  4
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   use_node_local_storage ....... False
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   wall_clock_breakdown ......... False
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   world_size ................... 64
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   zero_allow_untested_optimizer  False
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   zero_config .................. stage=0 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500000000 allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=False load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   zero_enabled ................. False
0: [2023-03-15 21:59:15,411] [INFO] [config.py:1011:print]   zero_optimization_stage ...... 0
0: [2023-03-15 21:59:15,411] [INFO] [config.py:996:print_user_config]   json = {
0:     "train_micro_batch_size_per_gpu": 4, 
0:     "train_batch_size": 256, 
0:     "gradient_clipping": 1.0, 
0:     "zero_optimization": {
0:         "stage": 0
0:     }, 
0:     "bf16": {
0:         "enabled": true
0:     }, 
0:     "steps_per_print": 2.000000e+03, 
0:     "wall_clock_breakdown": false
0: }
0: Time to load utils op: 0.00042557716369628906 seconds
0: [2023-03-15 21:59:15,412] [INFO] [engine.py:87:__init__] CONFIG: micro_batches=1 micro_batch_size=4
0: [2023-03-15 21:59:15,422] [INFO] [engine.py:145:__init__] RANK=0 STAGE=0 LAYERS=25 [0, 25) STAGE_PARAMS=280342528 (280.343M) TOTAL_PARAMS=280342528 (280.343M) UNIQUE_PARAMS=280342528 (280.343M)
6: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
4: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
6: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
4: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
3: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
2: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
3: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
3: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
2: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
4: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
6: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
3: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: WARNING: could not find the metadata file checkpoints_280m5b9400m 
1: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0:     will not load any checkpoints and will start from random
2: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
4: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
6: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
5: [2023-03-15 21:59:15,429] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
1: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
4: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
6: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
2: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
3: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
5: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
6: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
1: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
3: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
4: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
5: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
1: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
6: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
2: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
5: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
3: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
2: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
1: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
4: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
3: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
6: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
5: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
4: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
1: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
5: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
2: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
1: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
5: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
2: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
5: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
1: [2023-03-15 21:59:15,430] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_280m5b9400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: time (ms) | load-checkpoint: 6.53
0: estimated model parameters: 0.280342528
0: estimated model parameters without embeddings: 0.22673408
0: [after model, optimizer, and learning rate scheduler are built] datetime: 2023-03-15 21:59:15 
0: > building train, validation, and test datasets ...
0:  > datasets target sizes (minimum size):
0:     train:      2884878
0:     validation: 3072
0:     test:       256
0: > building train, validation, and test datasets for GPT ...
0:  > building dataset index ...
0:     reading sizes...
0:     reading pointers...
0:     reading document index...
0:     creating numpy buffer of mmap...
0:     creating memory view of numpy buffer...
0:  > finished creating indexed dataset in 0.006673 seconds
0:     number of documents: 835726
0:  > dataset split:
0:     train:
0:      document indices in [0, 835726) total of 835726 documents
0:  > WARNING: could not find index map files, building the indices on rank 0 ...
0:  > last epoch number of samples (153471) is smaller than 95.0% of number of samples per epoch (195100), setting separate_last_epoch to True
0:  > elasped time to build and save doc-idx mapping (seconds): 0.556262
0:     using:
0:      number of documents:       835726
0:      number of epochs:          15
0:      sequence length:           2048
0:      total number of samples:   2926508
0:  > elasped time to build and save sample-idx mapping (seconds): 0.071125
0:  > building shuffle index with split [0, 2731407) and [2731407, 2926508) ...
0:  > elasped time to build and save shuffle-idx mapping (seconds): 0.074524
0:  > loading doc-idx mapping from /scratch/project_462000119/data/c4_subsampled/gpt2tok_c4_en_400M_text_document_train_indexmap_2884878ns_2048sl_1234s_doc_idx.npy
0:  > loading sample-idx mapping from /scratch/project_462000119/data/c4_subsampled/gpt2tok_c4_en_400M_text_document_train_indexmap_2884878ns_2048sl_1234s_sample_idx.npy
0:  > loading shuffle-idx mapping from /scratch/project_462000119/data/c4_subsampled/gpt2tok_c4_en_400M_text_document_train_indexmap_2884878ns_2048sl_1234s_shuffle_idx.npy
0:     loaded indexed file in 0.012 seconds
0:     total number of samples: 2926509
0:     total number of epochs: 15
0:  > building dataset index ...
0:     reading sizes...
0:     reading pointers...
0:     reading document index...
0:     creating numpy buffer of mmap...
0:     creating memory view of numpy buffer...
0:  > finished creating indexed dataset in 0.044785 seconds
0:     number of documents: 364608
0:  > dataset split:
0:     validation:
0:      document indices in [0, 364608) total of 364608 documents
0:  > loading doc-idx mapping from /scratch/project_462000119/data/c4_validation/gpt2tok_c4validation_rerun_text_document_validation_indexmap_3072ns_2048sl_1234s_doc_idx.npy
0:  > loading sample-idx mapping from /scratch/project_462000119/data/c4_validation/gpt2tok_c4validation_rerun_text_document_validation_indexmap_3072ns_2048sl_1234s_sample_idx.npy
0:  > loading shuffle-idx mapping from /scratch/project_462000119/data/c4_validation/gpt2tok_c4validation_rerun_text_document_validation_indexmap_3072ns_2048sl_1234s_shuffle_idx.npy
0:     loaded indexed file in 0.075 seconds
0:     total number of samples: 84978
0:     total number of epochs: 1
0: > finished creating GPT datasets ...
0: [after dataloaders are built] datetime: 2023-03-15 21:59:29 
0: done with setup ...
0: training ...
0: Number of parameters: [tensor rank - pipeline rank] w/ and w/o embeddings:
7: time (ms) | model-and-optimizer-setup: 17015.35 | train/valid/test-data-iterators-setup: 13710.54
0: [000-000] 0.2803B / 0.2267B
0: [before the start of training step] datetime: 2023-03-15 21:59:29 
0: [Rank 0] (after 10 iterations) memory (MB) | allocated: 3765.8056640625 | max allocated: 34106.49267578125 | reserved: 35064.0 | max reserved: 35064.0
7:  iteration       10/   11269 | consumed samples:         2560 | consumed tokens:      5242880 | elapsed time per iteration (s): 1.36 | learning rate: 1.775E-05 | global batch size:   256 | lm loss: 1.027888E+01 | grad norm: 2.686 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 188.292 | TFLOPs: 12.31 |
7:  iteration       20/   11269 | consumed samples:         5120 | consumed tokens:     10485760 | elapsed time per iteration (s): 0.50 | learning rate: 3.550E-05 | global batch size:   256 | lm loss: 9.282013E+00 | grad norm: 1.882 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 507.128 | TFLOPs: 33.17 |
7:  iteration       30/   11269 | consumed samples:         7680 | consumed tokens:     15728640 | elapsed time per iteration (s): 0.50 | learning rate: 5.324E-05 | global batch size:   256 | lm loss: 8.801241E+00 | grad norm: 1.876 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 515.602 | TFLOPs: 33.72 |
7:  iteration       40/   11269 | consumed samples:        10240 | consumed tokens:     20971520 | elapsed time per iteration (s): 0.49 | learning rate: 7.099E-05 | global batch size:   256 | lm loss: 8.217269E+00 | grad norm: 1.086 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 522.455 | TFLOPs: 34.17 |
7:  iteration       50/   11269 | consumed samples:        12800 | consumed tokens:     26214400 | elapsed time per iteration (s): 0.49 | learning rate: 8.874E-05 | global batch size:   256 | lm loss: 7.653390E+00 | grad norm: 1.025 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 522.280 | TFLOPs: 34.16 |
7:  iteration       60/   11269 | consumed samples:        15360 | consumed tokens:     31457280 | elapsed time per iteration (s): 0.50 | learning rate: 1.065E-04 | global batch size:   256 | lm loss: 7.292472E+00 | grad norm: 0.727 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 507.201 | TFLOPs: 33.17 |
7:  iteration       70/   11269 | consumed samples:        17920 | consumed tokens:     36700160 | elapsed time per iteration (s): 0.49 | learning rate: 1.242E-04 | global batch size:   256 | lm loss: 7.099794E+00 | grad norm: 0.930 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 523.555 | TFLOPs: 34.24 |
7:  iteration       80/   11269 | consumed samples:        20480 | consumed tokens:     41943040 | elapsed time per iteration (s): 0.49 | learning rate: 1.420E-04 | global batch size:   256 | lm loss: 6.961439E+00 | grad norm: 0.813 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 527.797 | TFLOPs: 34.52 |
7:  iteration       90/   11269 | consumed samples:        23040 | consumed tokens:     47185920 | elapsed time per iteration (s): 0.49 | learning rate: 1.597E-04 | global batch size:   256 | lm loss: 6.817342E+00 | grad norm: 0.806 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 519.459 | TFLOPs: 33.97 |
7:  iteration      100/   11269 | consumed samples:        25600 | consumed tokens:     52428800 | elapsed time per iteration (s): 0.49 | learning rate: 1.775E-04 | global batch size:   256 | lm loss: 6.711210E+00 | grad norm: 0.722 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 520.806 | TFLOPs: 34.06 |
7:  iteration      110/   11269 | consumed samples:        28160 | consumed tokens:     57671680 | elapsed time per iteration (s): 0.50 | learning rate: 1.952E-04 | global batch size:   256 | lm loss: 6.613287E+00 | grad norm: 1.035 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 511.305 | TFLOPs: 33.44 |
7:  iteration      120/   11269 | consumed samples:        30720 | consumed tokens:     62914560 | elapsed time per iteration (s): 0.48 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 6.510976E+00 | grad norm: 0.393 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.614 | TFLOPs: 34.96 |
7:  iteration      130/   11269 | consumed samples:        33280 | consumed tokens:     68157440 | elapsed time per iteration (s): 0.48 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 6.440462E+00 | grad norm: 0.630 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.191 | TFLOPs: 35.20 |
7:  iteration      140/   11269 | consumed samples:        35840 | consumed tokens:     73400320 | elapsed time per iteration (s): 0.48 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 6.427649E+00 | grad norm: 0.554 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.410 | TFLOPs: 35.08 |
7:  iteration      150/   11269 | consumed samples:        38400 | consumed tokens:     78643200 | elapsed time per iteration (s): 0.48 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 6.363493E+00 | grad norm: 0.361 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.177 | TFLOPs: 35.00 |
7:  iteration      160/   11269 | consumed samples:        40960 | consumed tokens:     83886080 | elapsed time per iteration (s): 0.48 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 6.318335E+00 | grad norm: 0.548 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.664 | TFLOPs: 34.97 |
7:  iteration      170/   11269 | consumed samples:        43520 | consumed tokens:     89128960 | elapsed time per iteration (s): 0.49 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 6.267995E+00 | grad norm: 0.418 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 524.590 | TFLOPs: 34.31 |
7:  iteration      180/   11269 | consumed samples:        46080 | consumed tokens:     94371840 | elapsed time per iteration (s): 0.49 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 6.230944E+00 | grad norm: 0.383 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 527.443 | TFLOPs: 34.49 |
7:  iteration      190/   11269 | consumed samples:        48640 | consumed tokens:     99614720 | elapsed time per iteration (s): 0.48 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 6.213390E+00 | grad norm: 0.817 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 528.793 | TFLOPs: 34.58 |
7:  iteration      200/   11269 | consumed samples:        51200 | consumed tokens:    104857600 | elapsed time per iteration (s): 0.48 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 6.200788E+00 | grad norm: 0.446 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.983 | TFLOPs: 35.12 |
7:  iteration      210/   11269 | consumed samples:        53760 | consumed tokens:    110100480 | elapsed time per iteration (s): 0.47 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 6.147212E+00 | grad norm: 0.475 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.415 | TFLOPs: 35.47 |
7:  iteration      220/   11269 | consumed samples:        56320 | consumed tokens:    115343360 | elapsed time per iteration (s): 0.48 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 6.127728E+00 | grad norm: 0.483 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 532.287 | TFLOPs: 34.81 |
7:  iteration      230/   11269 | consumed samples:        58880 | consumed tokens:    120586240 | elapsed time per iteration (s): 0.49 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 6.097730E+00 | grad norm: 0.391 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 520.181 | TFLOPs: 34.02 |
7:  iteration      240/   11269 | consumed samples:        61440 | consumed tokens:    125829120 | elapsed time per iteration (s): 0.48 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 6.073207E+00 | grad norm: 0.532 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.134 | TFLOPs: 35.13 |
7:  iteration      250/   11269 | consumed samples:        64000 | consumed tokens:    131072000 | elapsed time per iteration (s): 0.48 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 6.033714E+00 | grad norm: 0.510 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.045 | TFLOPs: 34.93 |
7:  iteration      260/   11269 | consumed samples:        66560 | consumed tokens:    136314880 | elapsed time per iteration (s): 0.48 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 6.030340E+00 | grad norm: 0.915 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.036 | TFLOPs: 34.86 |
7:  iteration      270/   11269 | consumed samples:        69120 | consumed tokens:    141557760 | elapsed time per iteration (s): 0.48 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 6.006889E+00 | grad norm: 0.660 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.097 | TFLOPs: 35.06 |
7:  iteration      280/   11269 | consumed samples:        71680 | consumed tokens:    146800640 | elapsed time per iteration (s): 0.49 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 5.970426E+00 | grad norm: 0.948 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 524.005 | TFLOPs: 34.27 |
7:  iteration      290/   11269 | consumed samples:        74240 | consumed tokens:    152043520 | elapsed time per iteration (s): 0.49 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 5.940175E+00 | grad norm: 0.488 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 527.560 | TFLOPs: 34.50 |
7:  iteration      300/   11269 | consumed samples:        76800 | consumed tokens:    157286400 | elapsed time per iteration (s): 0.48 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 5.934694E+00 | grad norm: 0.710 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.748 | TFLOPs: 35.17 |
7:  iteration      310/   11269 | consumed samples:        79360 | consumed tokens:    162529280 | elapsed time per iteration (s): 0.49 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 5.894152E+00 | grad norm: 0.422 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 526.023 | TFLOPs: 34.40 |
7:  iteration      320/   11269 | consumed samples:        81920 | consumed tokens:    167772160 | elapsed time per iteration (s): 0.49 | learning rate: 1.998E-04 | global batch size:   256 | lm loss: 5.880116E+00 | grad norm: 0.517 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 523.796 | TFLOPs: 34.26 |
7:  iteration      330/   11269 | consumed samples:        84480 | consumed tokens:    173015040 | elapsed time per iteration (s): 0.48 | learning rate: 1.998E-04 | global batch size:   256 | lm loss: 5.829498E+00 | grad norm: 0.743 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.257 | TFLOPs: 34.87 |
7:  iteration      340/   11269 | consumed samples:        87040 | consumed tokens:    178257920 | elapsed time per iteration (s): 0.47 | learning rate: 1.998E-04 | global batch size:   256 | lm loss: 5.812987E+00 | grad norm: 0.638 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.109 | TFLOPs: 35.45 |
7:  iteration      350/   11269 | consumed samples:        89600 | consumed tokens:    183500800 | elapsed time per iteration (s): 0.50 | learning rate: 1.998E-04 | global batch size:   256 | lm loss: 5.778070E+00 | grad norm: 0.337 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 516.365 | TFLOPs: 33.77 |
7:  iteration      360/   11269 | consumed samples:        92160 | consumed tokens:    188743680 | elapsed time per iteration (s): 0.50 | learning rate: 1.998E-04 | global batch size:   256 | lm loss: 5.737553E+00 | grad norm: 0.814 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 512.925 | TFLOPs: 33.55 |
7:  iteration      370/   11269 | consumed samples:        94720 | consumed tokens:    193986560 | elapsed time per iteration (s): 0.49 | learning rate: 1.998E-04 | global batch size:   256 | lm loss: 5.730161E+00 | grad norm: 0.554 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 523.026 | TFLOPs: 34.21 |
7:  iteration      380/   11269 | consumed samples:        97280 | consumed tokens:    199229440 | elapsed time per iteration (s): 0.49 | learning rate: 1.997E-04 | global batch size:   256 | lm loss: 5.695152E+00 | grad norm: 0.731 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 523.541 | TFLOPs: 34.24 |
7:  iteration      390/   11269 | consumed samples:        99840 | consumed tokens:    204472320 | elapsed time per iteration (s): 0.48 | learning rate: 1.997E-04 | global batch size:   256 | lm loss: 5.674793E+00 | grad norm: 0.403 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 528.723 | TFLOPs: 34.58 |
7:  iteration      400/   11269 | consumed samples:       102400 | consumed tokens:    209715200 | elapsed time per iteration (s): 0.48 | learning rate: 1.997E-04 | global batch size:   256 | lm loss: 5.630848E+00 | grad norm: 0.989 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.640 | TFLOPs: 35.03 |
7:  iteration      410/   11269 | consumed samples:       104960 | consumed tokens:    214958080 | elapsed time per iteration (s): 0.50 | learning rate: 1.997E-04 | global batch size:   256 | lm loss: 5.635732E+00 | grad norm: 0.518 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 515.622 | TFLOPs: 33.72 |
7:  iteration      420/   11269 | consumed samples:       107520 | consumed tokens:    220200960 | elapsed time per iteration (s): 0.48 | learning rate: 1.997E-04 | global batch size:   256 | lm loss: 5.570592E+00 | grad norm: 0.483 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 529.920 | TFLOPs: 34.66 |
7:  iteration      430/   11269 | consumed samples:       110080 | consumed tokens:    225443840 | elapsed time per iteration (s): 0.48 | learning rate: 1.996E-04 | global batch size:   256 | lm loss: 5.554506E+00 | grad norm: 0.635 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 529.228 | TFLOPs: 34.61 |
7:  iteration      440/   11269 | consumed samples:       112640 | consumed tokens:    230686720 | elapsed time per iteration (s): 0.48 | learning rate: 1.996E-04 | global batch size:   256 | lm loss: 5.531062E+00 | grad norm: 0.866 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.936 | TFLOPs: 35.12 |
7:  iteration      450/   11269 | consumed samples:       115200 | consumed tokens:    235929600 | elapsed time per iteration (s): 0.48 | learning rate: 1.996E-04 | global batch size:   256 | lm loss: 5.511559E+00 | grad norm: 0.444 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.146 | TFLOPs: 34.87 |
7:  iteration      460/   11269 | consumed samples:       117760 | consumed tokens:    241172480 | elapsed time per iteration (s): 0.49 | learning rate: 1.996E-04 | global batch size:   256 | lm loss: 5.456729E+00 | grad norm: 0.916 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 525.116 | TFLOPs: 34.34 |
7:  iteration      470/   11269 | consumed samples:       120320 | consumed tokens:    246415360 | elapsed time per iteration (s): 0.49 | learning rate: 1.995E-04 | global batch size:   256 | lm loss: 5.479884E+00 | grad norm: 0.739 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 526.241 | TFLOPs: 34.42 |
7:  iteration      480/   11269 | consumed samples:       122880 | consumed tokens:    251658240 | elapsed time per iteration (s): 0.49 | learning rate: 1.995E-04 | global batch size:   256 | lm loss: 5.432080E+00 | grad norm: 0.485 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 521.080 | TFLOPs: 34.08 |
7:  iteration      490/   11269 | consumed samples:       125440 | consumed tokens:    256901120 | elapsed time per iteration (s): 0.49 | learning rate: 1.995E-04 | global batch size:   256 | lm loss: 5.394036E+00 | grad norm: 0.884 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 523.430 | TFLOPs: 34.23 |
7:  iteration      500/   11269 | consumed samples:       128000 | consumed tokens:    262144000 | elapsed time per iteration (s): 0.49 | learning rate: 1.995E-04 | global batch size:   256 | lm loss: 5.395821E+00 | grad norm: 0.671 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 522.905 | TFLOPs: 34.20 |
7:  iteration      510/   11269 | consumed samples:       130560 | consumed tokens:    267386880 | elapsed time per iteration (s): 0.48 | learning rate: 1.994E-04 | global batch size:   256 | lm loss: 5.347544E+00 | grad norm: 0.663 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.856 | TFLOPs: 35.18 |
7:  iteration      520/   11269 | consumed samples:       133120 | consumed tokens:    272629760 | elapsed time per iteration (s): 0.48 | learning rate: 1.994E-04 | global batch size:   256 | lm loss: 5.302108E+00 | grad norm: 0.572 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.402 | TFLOPs: 34.75 |
7:  iteration      530/   11269 | consumed samples:       135680 | consumed tokens:    277872640 | elapsed time per iteration (s): 0.48 | learning rate: 1.994E-04 | global batch size:   256 | lm loss: 5.293512E+00 | grad norm: 0.735 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.539 | TFLOPs: 34.96 |
7:  iteration      540/   11269 | consumed samples:       138240 | consumed tokens:    283115520 | elapsed time per iteration (s): 0.47 | learning rate: 1.993E-04 | global batch size:   256 | lm loss: 5.254497E+00 | grad norm: 0.618 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.572 | TFLOPs: 35.48 |
7:  iteration      550/   11269 | consumed samples:       140800 | consumed tokens:    288358400 | elapsed time per iteration (s): 0.48 | learning rate: 1.993E-04 | global batch size:   256 | lm loss: 5.264146E+00 | grad norm: 0.698 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.555 | TFLOPs: 35.03 |
7:  iteration      560/   11269 | consumed samples:       143360 | consumed tokens:    293601280 | elapsed time per iteration (s): 0.47 | learning rate: 1.993E-04 | global batch size:   256 | lm loss: 5.225822E+00 | grad norm: 0.853 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.069 | TFLOPs: 35.25 |
7:  iteration      570/   11269 | consumed samples:       145920 | consumed tokens:    298844160 | elapsed time per iteration (s): 0.47 | learning rate: 1.993E-04 | global batch size:   256 | lm loss: 5.209805E+00 | grad norm: 0.536 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.655 | TFLOPs: 35.42 |
7:  iteration      580/   11269 | consumed samples:       148480 | consumed tokens:    304087040 | elapsed time per iteration (s): 0.48 | learning rate: 1.992E-04 | global batch size:   256 | lm loss: 5.156770E+00 | grad norm: 0.809 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 529.189 | TFLOPs: 34.61 |
7:  iteration      590/   11269 | consumed samples:       151040 | consumed tokens:    309329920 | elapsed time per iteration (s): 0.48 | learning rate: 1.992E-04 | global batch size:   256 | lm loss: 5.151631E+00 | grad norm: 0.502 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 532.686 | TFLOPs: 34.84 |
7:  iteration      600/   11269 | consumed samples:       153600 | consumed tokens:    314572800 | elapsed time per iteration (s): 0.48 | learning rate: 1.992E-04 | global batch size:   256 | lm loss: 5.120596E+00 | grad norm: 0.741 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 530.133 | TFLOPs: 34.67 |
7:  iteration      610/   11269 | consumed samples:       156160 | consumed tokens:    319815680 | elapsed time per iteration (s): 0.48 | learning rate: 1.991E-04 | global batch size:   256 | lm loss: 5.125335E+00 | grad norm: 0.562 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.552 | TFLOPs: 35.16 |
7:  iteration      620/   11269 | consumed samples:       158720 | consumed tokens:    325058560 | elapsed time per iteration (s): 0.48 | learning rate: 1.991E-04 | global batch size:   256 | lm loss: 5.092756E+00 | grad norm: 0.771 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.164 | TFLOPs: 34.74 |
7:  iteration      630/   11269 | consumed samples:       161280 | consumed tokens:    330301440 | elapsed time per iteration (s): 0.48 | learning rate: 1.990E-04 | global batch size:   256 | lm loss: 5.059850E+00 | grad norm: 0.557 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.216 | TFLOPs: 34.94 |
7:  iteration      640/   11269 | consumed samples:       163840 | consumed tokens:    335544320 | elapsed time per iteration (s): 0.48 | learning rate: 1.990E-04 | global batch size:   256 | lm loss: 5.024911E+00 | grad norm: 0.548 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 528.566 | TFLOPs: 34.57 |
7:  iteration      650/   11269 | consumed samples:       166400 | consumed tokens:    340787200 | elapsed time per iteration (s): 0.48 | learning rate: 1.990E-04 | global batch size:   256 | lm loss: 5.003077E+00 | grad norm: 0.768 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 529.392 | TFLOPs: 34.62 |
7:  iteration      660/   11269 | consumed samples:       168960 | consumed tokens:    346030080 | elapsed time per iteration (s): 0.48 | learning rate: 1.989E-04 | global batch size:   256 | lm loss: 4.994066E+00 | grad norm: 0.683 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.957 | TFLOPs: 34.92 |
7:  iteration      670/   11269 | consumed samples:       171520 | consumed tokens:    351272960 | elapsed time per iteration (s): 0.48 | learning rate: 1.989E-04 | global batch size:   256 | lm loss: 4.976462E+00 | grad norm: 0.705 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.040 | TFLOPs: 34.86 |
7:  iteration      680/   11269 | consumed samples:       174080 | consumed tokens:    356515840 | elapsed time per iteration (s): 0.48 | learning rate: 1.989E-04 | global batch size:   256 | lm loss: 4.974440E+00 | grad norm: 0.871 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.984 | TFLOPs: 34.92 |
7:  iteration      690/   11269 | consumed samples:       176640 | consumed tokens:    361758720 | elapsed time per iteration (s): 0.48 | learning rate: 1.988E-04 | global batch size:   256 | lm loss: 4.922567E+00 | grad norm: 0.604 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.805 | TFLOPs: 35.11 |
7:  iteration      700/   11269 | consumed samples:       179200 | consumed tokens:    367001600 | elapsed time per iteration (s): 0.48 | learning rate: 1.988E-04 | global batch size:   256 | lm loss: 4.899709E+00 | grad norm: 0.824 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.813 | TFLOPs: 35.17 |
7:  iteration      710/   11269 | consumed samples:       181760 | consumed tokens:    372244480 | elapsed time per iteration (s): 0.48 | learning rate: 1.987E-04 | global batch size:   256 | lm loss: 4.885059E+00 | grad norm: 0.544 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.479 | TFLOPs: 34.89 |
7:  iteration      720/   11269 | consumed samples:       184320 | consumed tokens:    377487360 | elapsed time per iteration (s): 0.47 | learning rate: 1.987E-04 | global batch size:   256 | lm loss: 4.864639E+00 | grad norm: 1.157 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.433 | TFLOPs: 35.28 |
7:  iteration      730/   11269 | consumed samples:       186880 | consumed tokens:    382730240 | elapsed time per iteration (s): 0.48 | learning rate: 1.986E-04 | global batch size:   256 | lm loss: 4.845634E+00 | grad norm: 0.429 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.294 | TFLOPs: 35.14 |
7:  iteration      740/   11269 | consumed samples:       189440 | consumed tokens:    387973120 | elapsed time per iteration (s): 0.48 | learning rate: 1.986E-04 | global batch size:   256 | lm loss: 4.823149E+00 | grad norm: 0.994 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.043 | TFLOPs: 35.12 |
7:  iteration      750/   11269 | consumed samples:       192000 | consumed tokens:    393216000 | elapsed time per iteration (s): 0.48 | learning rate: 1.986E-04 | global batch size:   256 | lm loss: 4.803323E+00 | grad norm: 0.706 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.007 | TFLOPs: 34.92 |
7:  iteration      760/   11269 | consumed samples:       194560 | consumed tokens:    398458880 | elapsed time per iteration (s): 0.47 | learning rate: 1.985E-04 | global batch size:   256 | lm loss: 4.772005E+00 | grad norm: 0.678 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.781 | TFLOPs: 35.30 |
7:  iteration      770/   11269 | consumed samples:       197120 | consumed tokens:    403701760 | elapsed time per iteration (s): 0.48 | learning rate: 1.985E-04 | global batch size:   256 | lm loss: 4.717144E+00 | grad norm: 0.902 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 529.921 | TFLOPs: 34.66 |
7:  iteration      780/   11269 | consumed samples:       199680 | consumed tokens:    408944640 | elapsed time per iteration (s): 0.48 | learning rate: 1.984E-04 | global batch size:   256 | lm loss: 4.721557E+00 | grad norm: 0.667 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.553 | TFLOPs: 35.22 |
7:  iteration      790/   11269 | consumed samples:       202240 | consumed tokens:    414187520 | elapsed time per iteration (s): 0.48 | learning rate: 1.984E-04 | global batch size:   256 | lm loss: 4.679041E+00 | grad norm: 0.495 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.519 | TFLOPs: 34.96 |
7:  iteration      800/   11269 | consumed samples:       204800 | consumed tokens:    419430400 | elapsed time per iteration (s): 0.48 | learning rate: 1.983E-04 | global batch size:   256 | lm loss: 4.666647E+00 | grad norm: 0.545 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.097 | TFLOPs: 34.73 |
7:  iteration      810/   11269 | consumed samples:       207360 | consumed tokens:    424673280 | elapsed time per iteration (s): 0.48 | learning rate: 1.983E-04 | global batch size:   256 | lm loss: 4.636638E+00 | grad norm: 0.495 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.098 | TFLOPs: 34.93 |
7:  iteration      820/   11269 | consumed samples:       209920 | consumed tokens:    429916160 | elapsed time per iteration (s): 0.49 | learning rate: 1.982E-04 | global batch size:   256 | lm loss: 4.609793E+00 | grad norm: 0.642 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 523.860 | TFLOPs: 34.26 |
7:  iteration      830/   11269 | consumed samples:       212480 | consumed tokens:    435159040 | elapsed time per iteration (s): 0.48 | learning rate: 1.982E-04 | global batch size:   256 | lm loss: 4.595002E+00 | grad norm: 0.504 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.227 | TFLOPs: 35.13 |
7:  iteration      840/   11269 | consumed samples:       215040 | consumed tokens:    440401920 | elapsed time per iteration (s): 0.48 | learning rate: 1.981E-04 | global batch size:   256 | lm loss: 4.565396E+00 | grad norm: 0.535 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 530.411 | TFLOPs: 34.69 |
7:  iteration      850/   11269 | consumed samples:       217600 | consumed tokens:    445644800 | elapsed time per iteration (s): 0.48 | learning rate: 1.981E-04 | global batch size:   256 | lm loss: 4.560812E+00 | grad norm: 0.636 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 527.962 | TFLOPs: 34.53 |
7:  iteration      860/   11269 | consumed samples:       220160 | consumed tokens:    450887680 | elapsed time per iteration (s): 0.48 | learning rate: 1.980E-04 | global batch size:   256 | lm loss: 4.539445E+00 | grad norm: 0.562 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.014 | TFLOPs: 34.86 |
7:  iteration      870/   11269 | consumed samples:       222720 | consumed tokens:    456130560 | elapsed time per iteration (s): 0.48 | learning rate: 1.980E-04 | global batch size:   256 | lm loss: 4.551311E+00 | grad norm: 0.642 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.242 | TFLOPs: 34.87 |
7:  iteration      880/   11269 | consumed samples:       225280 | consumed tokens:    461373440 | elapsed time per iteration (s): 0.48 | learning rate: 1.979E-04 | global batch size:   256 | lm loss: 4.524503E+00 | grad norm: 0.552 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.371 | TFLOPs: 34.88 |
7:  iteration      890/   11269 | consumed samples:       227840 | consumed tokens:    466616320 | elapsed time per iteration (s): 0.48 | learning rate: 1.979E-04 | global batch size:   256 | lm loss: 4.513066E+00 | grad norm: 0.531 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 530.638 | TFLOPs: 34.70 |
7:  iteration      900/   11269 | consumed samples:       230400 | consumed tokens:    471859200 | elapsed time per iteration (s): 0.48 | learning rate: 1.978E-04 | global batch size:   256 | lm loss: 4.501746E+00 | grad norm: 0.513 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.479 | TFLOPs: 35.09 |
7:  iteration      910/   11269 | consumed samples:       232960 | consumed tokens:    477102080 | elapsed time per iteration (s): 0.48 | learning rate: 1.977E-04 | global batch size:   256 | lm loss: 4.470564E+00 | grad norm: 0.520 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.426 | TFLOPs: 35.02 |
7:  iteration      920/   11269 | consumed samples:       235520 | consumed tokens:    482344960 | elapsed time per iteration (s): 0.48 | learning rate: 1.977E-04 | global batch size:   256 | lm loss: 4.467780E+00 | grad norm: 0.668 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 532.448 | TFLOPs: 34.82 |
7:  iteration      930/   11269 | consumed samples:       238080 | consumed tokens:    487587840 | elapsed time per iteration (s): 0.48 | learning rate: 1.976E-04 | global batch size:   256 | lm loss: 4.458410E+00 | grad norm: 0.619 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 532.008 | TFLOPs: 34.79 |
7:  iteration      940/   11269 | consumed samples:       240640 | consumed tokens:    492830720 | elapsed time per iteration (s): 0.48 | learning rate: 1.976E-04 | global batch size:   256 | lm loss: 4.438409E+00 | grad norm: 0.677 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 532.956 | TFLOPs: 34.86 |
7:  iteration      950/   11269 | consumed samples:       243200 | consumed tokens:    498073600 | elapsed time per iteration (s): 0.48 | learning rate: 1.975E-04 | global batch size:   256 | lm loss: 4.428521E+00 | grad norm: 0.508 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.069 | TFLOPs: 34.86 |
7:  iteration      960/   11269 | consumed samples:       245760 | consumed tokens:    503316480 | elapsed time per iteration (s): 0.47 | learning rate: 1.975E-04 | global batch size:   256 | lm loss: 4.423767E+00 | grad norm: 0.504 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.466 | TFLOPs: 35.28 |
7:  iteration      970/   11269 | consumed samples:       248320 | consumed tokens:    508559360 | elapsed time per iteration (s): 0.48 | learning rate: 1.974E-04 | global batch size:   256 | lm loss: 4.396809E+00 | grad norm: 0.387 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.104 | TFLOPs: 34.73 |
7:  iteration      980/   11269 | consumed samples:       250880 | consumed tokens:    513802240 | elapsed time per iteration (s): 0.49 | learning rate: 1.973E-04 | global batch size:   256 | lm loss: 4.381861E+00 | grad norm: 0.656 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 520.344 | TFLOPs: 34.03 |
7:  iteration      990/   11269 | consumed samples:       253440 | consumed tokens:    519045120 | elapsed time per iteration (s): 0.48 | learning rate: 1.973E-04 | global batch size:   256 | lm loss: 4.388478E+00 | grad norm: 0.691 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 530.506 | TFLOPs: 34.69 |
7:  iteration     1000/   11269 | consumed samples:       256000 | consumed tokens:    524288000 | elapsed time per iteration (s): 0.47 | learning rate: 1.972E-04 | global batch size:   256 | lm loss: 4.382750E+00 | grad norm: 0.574 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.360 | TFLOPs: 35.47 |
7: -----------------------------------------------------------------------------------------------
7: validation loss at iteration 1000 | lm loss value: 4.359394E+00 | lm loss PPL: 7.820969E+01 | 
7: -----------------------------------------------------------------------------------------------
0: saving checkpoint at iteration    1000 to checkpoints_280m5b9400m
0: [2023-03-15 22:07:41,254] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step1000 is begin to save!
0: [2023-03-15 22:07:41,274] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_01-model_00-model_states.pt...
0: [2023-03-15 22:07:41,409] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_01-model_00-model_states.pt.
0: [2023-03-15 22:07:41,409] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_03-model_00-model_states.pt...
0: [2023-03-15 22:07:41,436] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_03-model_00-model_states.pt.
0: [2023-03-15 22:07:41,436] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_04-model_00-model_states.pt...
0: [2023-03-15 22:07:41,461] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_04-model_00-model_states.pt.
0: [2023-03-15 22:07:41,461] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_05-model_00-model_states.pt...
0: [2023-03-15 22:07:41,485] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_05-model_00-model_states.pt.
0: [2023-03-15 22:07:41,485] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_06-model_00-model_states.pt...
0: [2023-03-15 22:07:41,509] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_06-model_00-model_states.pt.
0: [2023-03-15 22:07:41,509] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_07-model_00-model_states.pt...
0: [2023-03-15 22:07:41,533] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_07-model_00-model_states.pt.
0: [2023-03-15 22:07:41,533] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_08-model_00-model_states.pt...
0: [2023-03-15 22:07:41,557] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_08-model_00-model_states.pt.
0: [2023-03-15 22:07:41,557] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_09-model_00-model_states.pt...
0: [2023-03-15 22:07:41,581] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_09-model_00-model_states.pt.
0: [2023-03-15 22:07:41,581] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_10-model_00-model_states.pt...
0: [2023-03-15 22:07:41,605] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_10-model_00-model_states.pt.
0: [2023-03-15 22:07:41,605] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_11-model_00-model_states.pt...
0: [2023-03-15 22:07:41,628] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_11-model_00-model_states.pt.
0: [2023-03-15 22:07:41,629] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_12-model_00-model_states.pt...
0: [2023-03-15 22:07:41,652] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_12-model_00-model_states.pt.
0: [2023-03-15 22:07:41,653] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_13-model_00-model_states.pt...
0: [2023-03-15 22:07:41,676] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_13-model_00-model_states.pt.
0: [2023-03-15 22:07:41,677] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_14-model_00-model_states.pt...
0: [2023-03-15 22:07:41,700] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_14-model_00-model_states.pt.
0: [2023-03-15 22:07:41,700] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_15-model_00-model_states.pt...
0: [2023-03-15 22:07:41,724] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_15-model_00-model_states.pt.
0: [2023-03-15 22:07:41,724] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_16-model_00-model_states.pt...
0: [2023-03-15 22:07:41,749] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_16-model_00-model_states.pt.
0: [2023-03-15 22:07:41,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_17-model_00-model_states.pt...
0: [2023-03-15 22:07:41,773] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_17-model_00-model_states.pt.
0: [2023-03-15 22:07:41,773] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_18-model_00-model_states.pt...
0: [2023-03-15 22:07:41,797] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_18-model_00-model_states.pt.
0: [2023-03-15 22:07:41,797] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_19-model_00-model_states.pt...
0: [2023-03-15 22:07:41,821] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_19-model_00-model_states.pt.
0: [2023-03-15 22:07:41,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_20-model_00-model_states.pt...
0: [2023-03-15 22:07:41,845] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_20-model_00-model_states.pt.
0: [2023-03-15 22:07:41,845] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/layer_22-model_00-model_states.pt...
0: [2023-03-15 22:07:41,846] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/layer_22-model_00-model_states.pt.
0: [2023-03-15 22:07:41,847] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_280m5b9400m/global_step1000/mp_rank_00_model_states.pt
0: [2023-03-15 22:07:41,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/mp_rank_00_model_states.pt...
0: [2023-03-15 22:07:41,850] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/mp_rank_00_model_states.pt.
0: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:07:41,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:07:41,868] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:07:41,924] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:07:41,927] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:07:41,928] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-15 22:07:41,928] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
0: [2023-03-15 22:07:41,935] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:07:41,935] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-15 22:07:41,935] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
0: [2023-03-15 22:07:41,935] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:07:41,936] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-15 22:07:41,936] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
1: [2023-03-15 22:07:41,936] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:07:41,936] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-15 22:07:41,936] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
5: [2023-03-15 22:07:41,949] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:07:41,957] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:07:41,957] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-15 22:07:41,957] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
5: [2023-03-15 22:07:41,949] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-15 22:07:41,949] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
5: [2023-03-15 22:07:41,949] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:07:41,949] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-15 22:07:41,949] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
5: [2023-03-15 22:07:41,950] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:07:41,950] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-15 22:07:41,950] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
5: [2023-03-15 22:07:41,959] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:07:41,959] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-15 22:07:41,959] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
5: [2023-03-15 22:07:41,959] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:07:41,959] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-15 22:07:41,959] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
5: [2023-03-15 22:07:41,959] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:07:41,959] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-15 22:07:41,959] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
5: [2023-03-15 22:07:41,959] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:07:41,959] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-15 22:07:41,959] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
1: [2023-03-15 22:07:41,977] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:07:41,977] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:07:41,977] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-15 22:07:41,977] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-15 22:07:41,977] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
1: [2023-03-15 22:07:41,978] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
4: [2023-03-15 22:07:41,945] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:07:41,945] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-15 22:07:41,945] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
4: [2023-03-15 22:07:41,945] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:07:41,945] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-15 22:07:41,945] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
4: [2023-03-15 22:07:41,945] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:07:41,945] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-15 22:07:41,945] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
4: [2023-03-15 22:07:41,945] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:07:41,946] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-15 22:07:41,946] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
4: [2023-03-15 22:07:41,953] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:07:41,953] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:07:41,953] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:07:41,953] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:07:41,954] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-15 22:07:41,954] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-15 22:07:41,954] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-15 22:07:41,954] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-15 22:07:41,954] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
4: [2023-03-15 22:07:41,954] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
4: [2023-03-15 22:07:41,954] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
4: [2023-03-15 22:07:41,954] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
0: [2023-03-15 22:07:41,986] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:07:41,986] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-15 22:07:41,986] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
1: [2023-03-15 22:07:41,987] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:07:41,987] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-15 22:07:41,987] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
1: [2023-03-15 22:07:41,987] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:07:41,987] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-15 22:07:41,988] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
7: [2023-03-15 22:07:41,989] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:07:41,989] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:07:41,989] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:07:41,990] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-15 22:07:41,990] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-15 22:07:41,990] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-15 22:07:41,990] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
7: [2023-03-15 22:07:41,990] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
7: [2023-03-15 22:07:41,990] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
2: [2023-03-15 22:07:41,990] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:07:41,990] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:07:41,990] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:07:41,990] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:07:41,990] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-15 22:07:41,990] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
5: [2023-03-15 22:07:42,001] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:07:42,001] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-15 22:07:42,001] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
7: [2023-03-15 22:07:42,004] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:07:42,004] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:07:42,004] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:07:42,004] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:07:42,004] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-15 22:07:42,004] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-15 22:07:42,004] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-15 22:07:42,004] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-15 22:07:42,004] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
7: [2023-03-15 22:07:42,004] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
7: [2023-03-15 22:07:42,004] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
7: [2023-03-15 22:07:42,004] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
6: [2023-03-15 22:07:42,005] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:07:42,005] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:07:42,005] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:07:42,005] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-15 22:07:42,005] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-15 22:07:42,005] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-15 22:07:42,006] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
6: [2023-03-15 22:07:42,006] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
6: [2023-03-15 22:07:42,006] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
3: [2023-03-15 22:07:42,007] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:07:42,007] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:07:42,007] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:07:42,007] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:07:42,007] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:07:42,007] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:07:42,008] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-15 22:07:42,008] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-15 22:07:42,008] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-15 22:07:42,008] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-15 22:07:42,008] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-15 22:07:42,008] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-15 22:07:42,008] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
3: [2023-03-15 22:07:42,008] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
3: [2023-03-15 22:07:42,008] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
3: [2023-03-15 22:07:42,008] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
3: [2023-03-15 22:07:42,008] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
3: [2023-03-15 22:07:42,008] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
7: [2023-03-15 22:07:42,009] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:07:42,009] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-15 22:07:42,009] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
0: [2023-03-15 22:07:42,012] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:07:42,012] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-15 22:07:42,012] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
0: [2023-03-15 22:07:42,015] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-15 22:07:42,016] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
6: [2023-03-15 22:07:42,017] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:07:42,018] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:07:42,018] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:07:42,018] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:07:42,018] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-15 22:07:42,018] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-15 22:07:42,018] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-15 22:07:42,018] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
6: [2023-03-15 22:07:42,018] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-15 22:07:42,018] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
6: [2023-03-15 22:07:42,018] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
6: [2023-03-15 22:07:42,018] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
0: [2023-03-15 22:07:42,022] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:07:42,022] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-15 22:07:42,022] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
3: [2023-03-15 22:07:42,029] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:07:42,029] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-15 22:07:42,029] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
1: [2023-03-15 22:07:42,031] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:07:42,031] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-15 22:07:42,031] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
2: [2023-03-15 22:07:41,990] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-15 22:07:41,990] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
2: [2023-03-15 22:07:41,990] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
2: [2023-03-15 22:07:41,990] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-15 22:07:41,991] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
2: [2023-03-15 22:07:41,991] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
2: [2023-03-15 22:07:41,991] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:07:41,992] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-15 22:07:41,992] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
2: [2023-03-15 22:07:42,000] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:07:42,000] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:07:42,000] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:07:42,000] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-15 22:07:42,000] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-15 22:07:42,000] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-15 22:07:42,000] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
2: [2023-03-15 22:07:42,000] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
2: [2023-03-15 22:07:42,000] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
1: [2023-03-15 22:07:42,037] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:07:42,037] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:07:42,037] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-15 22:07:42,037] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-15 22:07:42,037] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
1: [2023-03-15 22:07:42,037] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
3: [2023-03-15 22:07:42,047] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:07:42,048] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-15 22:07:42,048] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
6: [2023-03-15 22:07:42,047] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:07:42,047] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step1000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-15 22:07:42,047] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step1000 is ready now!
0:   successfully saved checkpoint at iteration    1000 to checkpoints_280m5b9400m
7: time (ms) | save-checkpoint: 800.32
7:  iteration     1010/   11269 | consumed samples:       258560 | consumed tokens:    529530880 | elapsed time per iteration (s): 0.57 | learning rate: 1.971E-04 | global batch size:   256 | lm loss: 4.367416E+00 | grad norm: 0.593 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 450.918 | TFLOPs: 29.49 |
7:  iteration     1020/   11269 | consumed samples:       261120 | consumed tokens:    534773760 | elapsed time per iteration (s): 0.50 | learning rate: 1.971E-04 | global batch size:   256 | lm loss: 4.358165E+00 | grad norm: 0.547 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 513.627 | TFLOPs: 33.59 |
7:  iteration     1030/   11269 | consumed samples:       263680 | consumed tokens:    540016640 | elapsed time per iteration (s): 0.47 | learning rate: 1.970E-04 | global batch size:   256 | lm loss: 4.347829E+00 | grad norm: 0.446 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.912 | TFLOPs: 35.44 |
7:  iteration     1040/   11269 | consumed samples:       266240 | consumed tokens:    545259520 | elapsed time per iteration (s): 0.48 | learning rate: 1.969E-04 | global batch size:   256 | lm loss: 4.331307E+00 | grad norm: 0.580 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 532.931 | TFLOPs: 34.85 |
7:  iteration     1050/   11269 | consumed samples:       268800 | consumed tokens:    550502400 | elapsed time per iteration (s): 0.47 | learning rate: 1.969E-04 | global batch size:   256 | lm loss: 4.316079E+00 | grad norm: 0.589 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.311 | TFLOPs: 35.34 |
7:  iteration     1060/   11269 | consumed samples:       271360 | consumed tokens:    555745280 | elapsed time per iteration (s): 0.47 | learning rate: 1.968E-04 | global batch size:   256 | lm loss: 4.317997E+00 | grad norm: 0.556 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.250 | TFLOPs: 35.46 |
7:  iteration     1070/   11269 | consumed samples:       273920 | consumed tokens:    560988160 | elapsed time per iteration (s): 0.48 | learning rate: 1.967E-04 | global batch size:   256 | lm loss: 4.297785E+00 | grad norm: 0.456 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.578 | TFLOPs: 35.09 |
7:  iteration     1080/   11269 | consumed samples:       276480 | consumed tokens:    566231040 | elapsed time per iteration (s): 0.48 | learning rate: 1.967E-04 | global batch size:   256 | lm loss: 4.309339E+00 | grad norm: 0.484 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.335 | TFLOPs: 35.21 |
7:  iteration     1090/   11269 | consumed samples:       279040 | consumed tokens:    571473920 | elapsed time per iteration (s): 0.48 | learning rate: 1.966E-04 | global batch size:   256 | lm loss: 4.307839E+00 | grad norm: 0.713 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.095 | TFLOPs: 35.06 |
7:  iteration     1100/   11269 | consumed samples:       281600 | consumed tokens:    576716800 | elapsed time per iteration (s): 0.47 | learning rate: 1.965E-04 | global batch size:   256 | lm loss: 4.285108E+00 | grad norm: 0.494 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.940 | TFLOPs: 35.31 |
7:  iteration     1110/   11269 | consumed samples:       284160 | consumed tokens:    581959680 | elapsed time per iteration (s): 0.48 | learning rate: 1.965E-04 | global batch size:   256 | lm loss: 4.273550E+00 | grad norm: 0.621 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.801 | TFLOPs: 34.98 |
7:  iteration     1120/   11269 | consumed samples:       286720 | consumed tokens:    587202560 | elapsed time per iteration (s): 0.47 | learning rate: 1.964E-04 | global batch size:   256 | lm loss: 4.256940E+00 | grad norm: 0.535 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.203 | TFLOPs: 35.46 |
7:  iteration     1130/   11269 | consumed samples:       289280 | consumed tokens:    592445440 | elapsed time per iteration (s): 0.47 | learning rate: 1.963E-04 | global batch size:   256 | lm loss: 4.249532E+00 | grad norm: 0.485 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.055 | TFLOPs: 35.25 |
7:  iteration     1140/   11269 | consumed samples:       291840 | consumed tokens:    597688320 | elapsed time per iteration (s): 0.48 | learning rate: 1.963E-04 | global batch size:   256 | lm loss: 4.260006E+00 | grad norm: 0.498 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.163 | TFLOPs: 35.20 |
7:  iteration     1150/   11269 | consumed samples:       294400 | consumed tokens:    602931200 | elapsed time per iteration (s): 0.47 | learning rate: 1.962E-04 | global batch size:   256 | lm loss: 4.235439E+00 | grad norm: 0.430 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.037 | TFLOPs: 35.25 |
7:  iteration     1160/   11269 | consumed samples:       296960 | consumed tokens:    608174080 | elapsed time per iteration (s): 0.48 | learning rate: 1.961E-04 | global batch size:   256 | lm loss: 4.260288E+00 | grad norm: 0.518 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.634 | TFLOPs: 35.16 |
7:  iteration     1170/   11269 | consumed samples:       299520 | consumed tokens:    613416960 | elapsed time per iteration (s): 0.48 | learning rate: 1.960E-04 | global batch size:   256 | lm loss: 4.223132E+00 | grad norm: 0.563 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.997 | TFLOPs: 34.92 |
7:  iteration     1180/   11269 | consumed samples:       302080 | consumed tokens:    618659840 | elapsed time per iteration (s): 0.47 | learning rate: 1.960E-04 | global batch size:   256 | lm loss: 4.223728E+00 | grad norm: 0.390 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.481 | TFLOPs: 35.28 |
7:  iteration     1190/   11269 | consumed samples:       304640 | consumed tokens:    623902720 | elapsed time per iteration (s): 0.48 | learning rate: 1.959E-04 | global batch size:   256 | lm loss: 4.221468E+00 | grad norm: 0.628 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.250 | TFLOPs: 35.14 |
7:  iteration     1200/   11269 | consumed samples:       307200 | consumed tokens:    629145600 | elapsed time per iteration (s): 0.48 | learning rate: 1.958E-04 | global batch size:   256 | lm loss: 4.199981E+00 | grad norm: 0.714 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.873 | TFLOPs: 35.11 |
7:  iteration     1210/   11269 | consumed samples:       309760 | consumed tokens:    634388480 | elapsed time per iteration (s): 0.47 | learning rate: 1.957E-04 | global batch size:   256 | lm loss: 4.200072E+00 | grad norm: 0.565 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.294 | TFLOPs: 35.47 |
7:  iteration     1220/   11269 | consumed samples:       312320 | consumed tokens:    639631360 | elapsed time per iteration (s): 0.48 | learning rate: 1.957E-04 | global batch size:   256 | lm loss: 4.215726E+00 | grad norm: 0.681 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.432 | TFLOPs: 35.21 |
7:  iteration     1230/   11269 | consumed samples:       314880 | consumed tokens:    644874240 | elapsed time per iteration (s): 0.48 | learning rate: 1.956E-04 | global batch size:   256 | lm loss: 4.202612E+00 | grad norm: 0.593 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.818 | TFLOPs: 34.78 |
7:  iteration     1240/   11269 | consumed samples:       317440 | consumed tokens:    650117120 | elapsed time per iteration (s): 0.49 | learning rate: 1.955E-04 | global batch size:   256 | lm loss: 4.179997E+00 | grad norm: 0.352 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 525.535 | TFLOPs: 34.37 |
7:  iteration     1250/   11269 | consumed samples:       320000 | consumed tokens:    655360000 | elapsed time per iteration (s): 0.48 | learning rate: 1.954E-04 | global batch size:   256 | lm loss: 4.171236E+00 | grad norm: 0.555 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.758 | TFLOPs: 35.10 |
7:  iteration     1260/   11269 | consumed samples:       322560 | consumed tokens:    660602880 | elapsed time per iteration (s): 0.48 | learning rate: 1.953E-04 | global batch size:   256 | lm loss: 4.170303E+00 | grad norm: 0.418 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.754 | TFLOPs: 35.10 |
7:  iteration     1270/   11269 | consumed samples:       325120 | consumed tokens:    665845760 | elapsed time per iteration (s): 0.48 | learning rate: 1.953E-04 | global batch size:   256 | lm loss: 4.161396E+00 | grad norm: 0.395 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.024 | TFLOPs: 35.19 |
7:  iteration     1280/   11269 | consumed samples:       327680 | consumed tokens:    671088640 | elapsed time per iteration (s): 0.47 | learning rate: 1.952E-04 | global batch size:   256 | lm loss: 4.161057E+00 | grad norm: 0.503 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.117 | TFLOPs: 35.32 |
7:  iteration     1290/   11269 | consumed samples:       330240 | consumed tokens:    676331520 | elapsed time per iteration (s): 0.48 | learning rate: 1.951E-04 | global batch size:   256 | lm loss: 4.150000E+00 | grad norm: 0.531 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.055 | TFLOPs: 34.93 |
7:  iteration     1300/   11269 | consumed samples:       332800 | consumed tokens:    681574400 | elapsed time per iteration (s): 0.48 | learning rate: 1.950E-04 | global batch size:   256 | lm loss: 4.151115E+00 | grad norm: 0.474 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.995 | TFLOPs: 35.18 |
7:  iteration     1310/   11269 | consumed samples:       335360 | consumed tokens:    686817280 | elapsed time per iteration (s): 0.47 | learning rate: 1.949E-04 | global batch size:   256 | lm loss: 4.140461E+00 | grad norm: 0.435 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.208 | TFLOPs: 35.33 |
7:  iteration     1320/   11269 | consumed samples:       337920 | consumed tokens:    692060160 | elapsed time per iteration (s): 0.48 | learning rate: 1.948E-04 | global batch size:   256 | lm loss: 4.134912E+00 | grad norm: 0.487 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.470 | TFLOPs: 34.95 |
7:  iteration     1330/   11269 | consumed samples:       340480 | consumed tokens:    697303040 | elapsed time per iteration (s): 0.48 | learning rate: 1.948E-04 | global batch size:   256 | lm loss: 4.130267E+00 | grad norm: 0.439 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.866 | TFLOPs: 35.05 |
7:  iteration     1340/   11269 | consumed samples:       343040 | consumed tokens:    702545920 | elapsed time per iteration (s): 0.48 | learning rate: 1.947E-04 | global batch size:   256 | lm loss: 4.129687E+00 | grad norm: 0.449 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.526 | TFLOPs: 35.22 |
7:  iteration     1350/   11269 | consumed samples:       345600 | consumed tokens:    707788800 | elapsed time per iteration (s): 0.47 | learning rate: 1.946E-04 | global batch size:   256 | lm loss: 4.121521E+00 | grad norm: 0.496 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.818 | TFLOPs: 35.30 |
7:  iteration     1360/   11269 | consumed samples:       348160 | consumed tokens:    713031680 | elapsed time per iteration (s): 0.47 | learning rate: 1.945E-04 | global batch size:   256 | lm loss: 4.109702E+00 | grad norm: 0.462 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.917 | TFLOPs: 35.31 |
7:  iteration     1370/   11269 | consumed samples:       350720 | consumed tokens:    718274560 | elapsed time per iteration (s): 0.48 | learning rate: 1.944E-04 | global batch size:   256 | lm loss: 4.102767E+00 | grad norm: 0.514 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.286 | TFLOPs: 34.88 |
7:  iteration     1380/   11269 | consumed samples:       353280 | consumed tokens:    723517440 | elapsed time per iteration (s): 0.48 | learning rate: 1.943E-04 | global batch size:   256 | lm loss: 4.118284E+00 | grad norm: 0.440 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.883 | TFLOPs: 34.78 |
7:  iteration     1390/   11269 | consumed samples:       355840 | consumed tokens:    728760320 | elapsed time per iteration (s): 0.47 | learning rate: 1.942E-04 | global batch size:   256 | lm loss: 4.100820E+00 | grad norm: 0.475 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.566 | TFLOPs: 35.29 |
7:  iteration     1400/   11269 | consumed samples:       358400 | consumed tokens:    734003200 | elapsed time per iteration (s): 0.47 | learning rate: 1.942E-04 | global batch size:   256 | lm loss: 4.088197E+00 | grad norm: 0.518 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.137 | TFLOPs: 35.46 |
7:  iteration     1410/   11269 | consumed samples:       360960 | consumed tokens:    739246080 | elapsed time per iteration (s): 0.48 | learning rate: 1.941E-04 | global batch size:   256 | lm loss: 4.090203E+00 | grad norm: 0.423 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.730 | TFLOPs: 35.17 |
7:  iteration     1420/   11269 | consumed samples:       363520 | consumed tokens:    744488960 | elapsed time per iteration (s): 0.48 | learning rate: 1.940E-04 | global batch size:   256 | lm loss: 4.095140E+00 | grad norm: 0.405 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.283 | TFLOPs: 35.14 |
7:  iteration     1430/   11269 | consumed samples:       366080 | consumed tokens:    749731840 | elapsed time per iteration (s): 0.48 | learning rate: 1.939E-04 | global batch size:   256 | lm loss: 4.086061E+00 | grad norm: 0.493 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 532.404 | TFLOPs: 34.82 |
7:  iteration     1440/   11269 | consumed samples:       368640 | consumed tokens:    754974720 | elapsed time per iteration (s): 0.48 | learning rate: 1.938E-04 | global batch size:   256 | lm loss: 4.084681E+00 | grad norm: 0.459 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.607 | TFLOPs: 35.16 |
7:  iteration     1450/   11269 | consumed samples:       371200 | consumed tokens:    760217600 | elapsed time per iteration (s): 0.48 | learning rate: 1.937E-04 | global batch size:   256 | lm loss: 4.078041E+00 | grad norm: 0.476 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.064 | TFLOPs: 35.12 |
7:  iteration     1460/   11269 | consumed samples:       373760 | consumed tokens:    765460480 | elapsed time per iteration (s): 0.47 | learning rate: 1.936E-04 | global batch size:   256 | lm loss: 4.074731E+00 | grad norm: 0.482 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.812 | TFLOPs: 35.43 |
7:  iteration     1470/   11269 | consumed samples:       376320 | consumed tokens:    770703360 | elapsed time per iteration (s): 0.47 | learning rate: 1.935E-04 | global batch size:   256 | lm loss: 4.075573E+00 | grad norm: 0.542 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.372 | TFLOPs: 35.27 |
7:  iteration     1480/   11269 | consumed samples:       378880 | consumed tokens:    775946240 | elapsed time per iteration (s): 0.47 | learning rate: 1.934E-04 | global batch size:   256 | lm loss: 4.066193E+00 | grad norm: 0.446 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.876 | TFLOPs: 35.44 |
7:  iteration     1490/   11269 | consumed samples:       381440 | consumed tokens:    781189120 | elapsed time per iteration (s): 0.49 | learning rate: 1.933E-04 | global batch size:   256 | lm loss: 4.063973E+00 | grad norm: 0.567 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 522.913 | TFLOPs: 34.20 |
7:  iteration     1500/   11269 | consumed samples:       384000 | consumed tokens:    786432000 | elapsed time per iteration (s): 0.48 | learning rate: 1.932E-04 | global batch size:   256 | lm loss: 4.055220E+00 | grad norm: 0.389 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.513 | TFLOPs: 35.09 |
7:  iteration     1510/   11269 | consumed samples:       386560 | consumed tokens:    791674880 | elapsed time per iteration (s): 0.48 | learning rate: 1.931E-04 | global batch size:   256 | lm loss: 4.041025E+00 | grad norm: 0.565 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.883 | TFLOPs: 35.24 |
7:  iteration     1520/   11269 | consumed samples:       389120 | consumed tokens:    796917760 | elapsed time per iteration (s): 0.47 | learning rate: 1.930E-04 | global batch size:   256 | lm loss: 4.062443E+00 | grad norm: 0.551 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.702 | TFLOPs: 35.43 |
7:  iteration     1530/   11269 | consumed samples:       391680 | consumed tokens:    802160640 | elapsed time per iteration (s): 0.48 | learning rate: 1.929E-04 | global batch size:   256 | lm loss: 4.047706E+00 | grad norm: 0.477 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.693 | TFLOPs: 35.16 |
7:  iteration     1540/   11269 | consumed samples:       394240 | consumed tokens:    807403520 | elapsed time per iteration (s): 0.48 | learning rate: 1.928E-04 | global batch size:   256 | lm loss: 4.044231E+00 | grad norm: 0.437 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.798 | TFLOPs: 34.98 |
7:  iteration     1550/   11269 | consumed samples:       396800 | consumed tokens:    812646400 | elapsed time per iteration (s): 0.48 | learning rate: 1.927E-04 | global batch size:   256 | lm loss: 4.043360E+00 | grad norm: 0.431 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.319 | TFLOPs: 35.14 |
7:  iteration     1560/   11269 | consumed samples:       399360 | consumed tokens:    817889280 | elapsed time per iteration (s): 0.48 | learning rate: 1.926E-04 | global batch size:   256 | lm loss: 4.011599E+00 | grad norm: 0.372 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.178 | TFLOPs: 35.07 |
7:  iteration     1570/   11269 | consumed samples:       401920 | consumed tokens:    823132160 | elapsed time per iteration (s): 0.48 | learning rate: 1.925E-04 | global batch size:   256 | lm loss: 4.019790E+00 | grad norm: 0.469 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.085 | TFLOPs: 35.19 |
7:  iteration     1580/   11269 | consumed samples:       404480 | consumed tokens:    828375040 | elapsed time per iteration (s): 0.47 | learning rate: 1.924E-04 | global batch size:   256 | lm loss: 4.018644E+00 | grad norm: 0.421 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.580 | TFLOPs: 35.42 |
7:  iteration     1590/   11269 | consumed samples:       407040 | consumed tokens:    833617920 | elapsed time per iteration (s): 0.48 | learning rate: 1.923E-04 | global batch size:   256 | lm loss: 4.010510E+00 | grad norm: 0.440 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.866 | TFLOPs: 34.91 |
7:  iteration     1600/   11269 | consumed samples:       409600 | consumed tokens:    838860800 | elapsed time per iteration (s): 0.48 | learning rate: 1.922E-04 | global batch size:   256 | lm loss: 3.996565E+00 | grad norm: 0.375 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.242 | TFLOPs: 35.20 |
7:  iteration     1610/   11269 | consumed samples:       412160 | consumed tokens:    844103680 | elapsed time per iteration (s): 0.47 | learning rate: 1.921E-04 | global batch size:   256 | lm loss: 4.002259E+00 | grad norm: 0.496 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.756 | TFLOPs: 35.37 |
7:  iteration     1620/   11269 | consumed samples:       414720 | consumed tokens:    849346560 | elapsed time per iteration (s): 0.48 | learning rate: 1.920E-04 | global batch size:   256 | lm loss: 4.001208E+00 | grad norm: 0.418 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.898 | TFLOPs: 34.98 |
7:  iteration     1630/   11269 | consumed samples:       417280 | consumed tokens:    854589440 | elapsed time per iteration (s): 0.48 | learning rate: 1.919E-04 | global batch size:   256 | lm loss: 3.987079E+00 | grad norm: 0.431 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.240 | TFLOPs: 34.94 |
7:  iteration     1640/   11269 | consumed samples:       419840 | consumed tokens:    859832320 | elapsed time per iteration (s): 0.48 | learning rate: 1.918E-04 | global batch size:   256 | lm loss: 4.000662E+00 | grad norm: 0.428 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.247 | TFLOPs: 34.94 |
7:  iteration     1650/   11269 | consumed samples:       422400 | consumed tokens:    865075200 | elapsed time per iteration (s): 0.48 | learning rate: 1.917E-04 | global batch size:   256 | lm loss: 4.002419E+00 | grad norm: 0.464 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.661 | TFLOPs: 35.16 |
7:  iteration     1660/   11269 | consumed samples:       424960 | consumed tokens:    870318080 | elapsed time per iteration (s): 0.48 | learning rate: 1.916E-04 | global batch size:   256 | lm loss: 3.987616E+00 | grad norm: 0.491 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.682 | TFLOPs: 35.16 |
7:  iteration     1670/   11269 | consumed samples:       427520 | consumed tokens:    875560960 | elapsed time per iteration (s): 0.48 | learning rate: 1.915E-04 | global batch size:   256 | lm loss: 3.985016E+00 | grad norm: 0.391 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.055 | TFLOPs: 34.86 |
7:  iteration     1680/   11269 | consumed samples:       430080 | consumed tokens:    880803840 | elapsed time per iteration (s): 0.47 | learning rate: 1.914E-04 | global batch size:   256 | lm loss: 3.978141E+00 | grad norm: 0.500 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.052 | TFLOPs: 35.45 |
7:  iteration     1690/   11269 | consumed samples:       432640 | consumed tokens:    886046720 | elapsed time per iteration (s): 0.47 | learning rate: 1.913E-04 | global batch size:   256 | lm loss: 3.965922E+00 | grad norm: 0.419 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.765 | TFLOPs: 35.43 |
7:  iteration     1700/   11269 | consumed samples:       435200 | consumed tokens:    891289600 | elapsed time per iteration (s): 0.48 | learning rate: 1.912E-04 | global batch size:   256 | lm loss: 3.969548E+00 | grad norm: 0.416 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 529.608 | TFLOPs: 34.64 |
7:  iteration     1710/   11269 | consumed samples:       437760 | consumed tokens:    896532480 | elapsed time per iteration (s): 0.47 | learning rate: 1.910E-04 | global batch size:   256 | lm loss: 3.959496E+00 | grad norm: 0.439 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.551 | TFLOPs: 35.29 |
7:  iteration     1720/   11269 | consumed samples:       440320 | consumed tokens:    901775360 | elapsed time per iteration (s): 0.48 | learning rate: 1.909E-04 | global batch size:   256 | lm loss: 3.950530E+00 | grad norm: 0.495 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.559 | TFLOPs: 35.22 |
7:  iteration     1730/   11269 | consumed samples:       442880 | consumed tokens:    907018240 | elapsed time per iteration (s): 0.48 | learning rate: 1.908E-04 | global batch size:   256 | lm loss: 3.957402E+00 | grad norm: 0.425 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.389 | TFLOPs: 35.15 |
7:  iteration     1740/   11269 | consumed samples:       445440 | consumed tokens:    912261120 | elapsed time per iteration (s): 0.48 | learning rate: 1.907E-04 | global batch size:   256 | lm loss: 3.963531E+00 | grad norm: 0.472 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.673 | TFLOPs: 35.23 |
7:  iteration     1750/   11269 | consumed samples:       448000 | consumed tokens:    917504000 | elapsed time per iteration (s): 0.48 | learning rate: 1.906E-04 | global batch size:   256 | lm loss: 3.958447E+00 | grad norm: 0.381 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.423 | TFLOPs: 35.08 |
7:  iteration     1760/   11269 | consumed samples:       450560 | consumed tokens:    922746880 | elapsed time per iteration (s): 0.49 | learning rate: 1.905E-04 | global batch size:   256 | lm loss: 3.941109E+00 | grad norm: 0.456 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 527.712 | TFLOPs: 34.51 |
7:  iteration     1770/   11269 | consumed samples:       453120 | consumed tokens:    927989760 | elapsed time per iteration (s): 0.50 | learning rate: 1.904E-04 | global batch size:   256 | lm loss: 3.949590E+00 | grad norm: 0.426 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 511.991 | TFLOPs: 33.48 |
7:  iteration     1780/   11269 | consumed samples:       455680 | consumed tokens:    933232640 | elapsed time per iteration (s): 0.48 | learning rate: 1.903E-04 | global batch size:   256 | lm loss: 3.928989E+00 | grad norm: 0.443 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 532.474 | TFLOPs: 34.82 |
7:  iteration     1790/   11269 | consumed samples:       458240 | consumed tokens:    938475520 | elapsed time per iteration (s): 0.48 | learning rate: 1.901E-04 | global batch size:   256 | lm loss: 3.931721E+00 | grad norm: 0.456 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.513 | TFLOPs: 35.22 |
7:  iteration     1800/   11269 | consumed samples:       460800 | consumed tokens:    943718400 | elapsed time per iteration (s): 0.48 | learning rate: 1.900E-04 | global batch size:   256 | lm loss: 3.926086E+00 | grad norm: 0.436 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.562 | TFLOPs: 35.22 |
7:  iteration     1810/   11269 | consumed samples:       463360 | consumed tokens:    948961280 | elapsed time per iteration (s): 0.48 | learning rate: 1.899E-04 | global batch size:   256 | lm loss: 3.928977E+00 | grad norm: 0.434 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 532.281 | TFLOPs: 34.81 |
7:  iteration     1820/   11269 | consumed samples:       465920 | consumed tokens:    954204160 | elapsed time per iteration (s): 0.47 | learning rate: 1.898E-04 | global batch size:   256 | lm loss: 3.939436E+00 | grad norm: 0.429 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.582 | TFLOPs: 35.42 |
7:  iteration     1830/   11269 | consumed samples:       468480 | consumed tokens:    959447040 | elapsed time per iteration (s): 0.48 | learning rate: 1.897E-04 | global batch size:   256 | lm loss: 3.920950E+00 | grad norm: 0.469 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.510 | TFLOPs: 35.22 |
7:  iteration     1840/   11269 | consumed samples:       471040 | consumed tokens:    964689920 | elapsed time per iteration (s): 0.48 | learning rate: 1.896E-04 | global batch size:   256 | lm loss: 3.932523E+00 | grad norm: 0.383 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.223 | TFLOPs: 35.07 |
7:  iteration     1850/   11269 | consumed samples:       473600 | consumed tokens:    969932800 | elapsed time per iteration (s): 0.48 | learning rate: 1.894E-04 | global batch size:   256 | lm loss: 3.927012E+00 | grad norm: 0.411 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.161 | TFLOPs: 34.93 |
7:  iteration     1860/   11269 | consumed samples:       476160 | consumed tokens:    975175680 | elapsed time per iteration (s): 0.48 | learning rate: 1.893E-04 | global batch size:   256 | lm loss: 3.912764E+00 | grad norm: 0.384 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 532.139 | TFLOPs: 34.80 |
7:  iteration     1870/   11269 | consumed samples:       478720 | consumed tokens:    980418560 | elapsed time per iteration (s): 0.48 | learning rate: 1.892E-04 | global batch size:   256 | lm loss: 3.915350E+00 | grad norm: 0.431 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.427 | TFLOPs: 35.21 |
7:  iteration     1880/   11269 | consumed samples:       481280 | consumed tokens:    985661440 | elapsed time per iteration (s): 0.48 | learning rate: 1.891E-04 | global batch size:   256 | lm loss: 3.920081E+00 | grad norm: 0.392 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.506 | TFLOPs: 35.22 |
7:  iteration     1890/   11269 | consumed samples:       483840 | consumed tokens:    990904320 | elapsed time per iteration (s): 0.49 | learning rate: 1.890E-04 | global batch size:   256 | lm loss: 3.905660E+00 | grad norm: 0.413 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 526.902 | TFLOPs: 34.46 |
7:  iteration     1900/   11269 | consumed samples:       486400 | consumed tokens:    996147200 | elapsed time per iteration (s): 0.47 | learning rate: 1.888E-04 | global batch size:   256 | lm loss: 3.901535E+00 | grad norm: 0.406 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.985 | TFLOPs: 35.45 |
7:  iteration     1910/   11269 | consumed samples:       488960 | consumed tokens:   1001390080 | elapsed time per iteration (s): 0.48 | learning rate: 1.887E-04 | global batch size:   256 | lm loss: 3.896963E+00 | grad norm: 0.442 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.549 | TFLOPs: 35.02 |
7:  iteration     1920/   11269 | consumed samples:       491520 | consumed tokens:   1006632960 | elapsed time per iteration (s): 0.48 | learning rate: 1.886E-04 | global batch size:   256 | lm loss: 3.910717E+00 | grad norm: 0.545 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.075 | TFLOPs: 35.19 |
7:  iteration     1930/   11269 | consumed samples:       494080 | consumed tokens:   1011875840 | elapsed time per iteration (s): 0.47 | learning rate: 1.885E-04 | global batch size:   256 | lm loss: 3.892570E+00 | grad norm: 0.454 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.448 | TFLOPs: 35.41 |
7:  iteration     1940/   11269 | consumed samples:       496640 | consumed tokens:   1017118720 | elapsed time per iteration (s): 0.47 | learning rate: 1.883E-04 | global batch size:   256 | lm loss: 3.906018E+00 | grad norm: 0.384 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.999 | TFLOPs: 35.25 |
7:  iteration     1950/   11269 | consumed samples:       499200 | consumed tokens:   1022361600 | elapsed time per iteration (s): 0.48 | learning rate: 1.882E-04 | global batch size:   256 | lm loss: 3.884351E+00 | grad norm: 0.464 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.044 | TFLOPs: 34.86 |
7:  iteration     1960/   11269 | consumed samples:       501760 | consumed tokens:   1027604480 | elapsed time per iteration (s): 0.48 | learning rate: 1.881E-04 | global batch size:   256 | lm loss: 3.902623E+00 | grad norm: 0.377 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.128 | TFLOPs: 35.13 |
7:  iteration     1970/   11269 | consumed samples:       504320 | consumed tokens:   1032847360 | elapsed time per iteration (s): 0.48 | learning rate: 1.880E-04 | global batch size:   256 | lm loss: 3.882310E+00 | grad norm: 0.761 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.016 | TFLOPs: 34.99 |
7:  iteration     1980/   11269 | consumed samples:       506880 | consumed tokens:   1038090240 | elapsed time per iteration (s): 0.47 | learning rate: 1.878E-04 | global batch size:   256 | lm loss: 3.912272E+00 | grad norm: 0.583 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.733 | TFLOPs: 35.36 |
7:  iteration     1990/   11269 | consumed samples:       509440 | consumed tokens:   1043333120 | elapsed time per iteration (s): 0.48 | learning rate: 1.877E-04 | global batch size:   256 | lm loss: 3.903329E+00 | grad norm: 0.443 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.767 | TFLOPs: 34.78 |
0: [2023-03-15 22:15:39,143] [INFO] [logging.py:68:log_dist] [Rank 0] step=2000, skipped=0, lr=[0.00018758615112345138, 0.00018758615112345138, 0.00018758615112345138], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration     2000/   11269 | consumed samples:       512000 | consumed tokens:   1048576000 | elapsed time per iteration (s): 0.47 | learning rate: 1.876E-04 | global batch size:   256 | lm loss: 3.884674E+00 | grad norm: 0.374 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.859 | TFLOPs: 35.44 |
0: steps: 2000 loss: 3.8585 iter time (s): 0.482 samples/sec: 531.170
7: -----------------------------------------------------------------------------------------------
7: validation loss at iteration 2000 | lm loss value: 3.851890E+00 | lm loss PPL: 4.708195E+01 | 
7: -----------------------------------------------------------------------------------------------
0: saving checkpoint at iteration    2000 to checkpoints_280m5b9400m
0: [2023-03-15 22:15:39,321] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step2000 is begin to save!
0: [2023-03-15 22:15:39,324] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_01-model_00-model_states.pt...
0: [2023-03-15 22:15:39,438] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_01-model_00-model_states.pt.
0: [2023-03-15 22:15:39,438] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_03-model_00-model_states.pt...
0: [2023-03-15 22:15:39,463] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_03-model_00-model_states.pt.
0: [2023-03-15 22:15:39,464] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_04-model_00-model_states.pt...
0: [2023-03-15 22:15:39,487] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_04-model_00-model_states.pt.
0: [2023-03-15 22:15:39,488] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_05-model_00-model_states.pt...
0: [2023-03-15 22:15:39,512] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_05-model_00-model_states.pt.
0: [2023-03-15 22:15:39,512] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_06-model_00-model_states.pt...
0: [2023-03-15 22:15:39,536] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_06-model_00-model_states.pt.
0: [2023-03-15 22:15:39,537] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_07-model_00-model_states.pt...
0: [2023-03-15 22:15:39,562] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_07-model_00-model_states.pt.
0: [2023-03-15 22:15:39,562] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_08-model_00-model_states.pt...
0: [2023-03-15 22:15:39,586] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_08-model_00-model_states.pt.
0: [2023-03-15 22:15:39,586] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_09-model_00-model_states.pt...
0: [2023-03-15 22:15:39,611] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_09-model_00-model_states.pt.
0: [2023-03-15 22:15:39,611] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_10-model_00-model_states.pt...
0: [2023-03-15 22:15:39,635] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_10-model_00-model_states.pt.
0: [2023-03-15 22:15:39,635] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_11-model_00-model_states.pt...
0: [2023-03-15 22:15:39,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_11-model_00-model_states.pt.
0: [2023-03-15 22:15:39,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_12-model_00-model_states.pt...
0: [2023-03-15 22:15:39,684] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_12-model_00-model_states.pt.
0: [2023-03-15 22:15:39,684] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_13-model_00-model_states.pt...
0: [2023-03-15 22:15:39,708] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_13-model_00-model_states.pt.
0: [2023-03-15 22:15:39,708] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_14-model_00-model_states.pt...
0: [2023-03-15 22:15:39,732] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_14-model_00-model_states.pt.
0: [2023-03-15 22:15:39,733] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_15-model_00-model_states.pt...
0: [2023-03-15 22:15:39,757] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_15-model_00-model_states.pt.
0: [2023-03-15 22:15:39,757] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_16-model_00-model_states.pt...
0: [2023-03-15 22:15:39,781] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_16-model_00-model_states.pt.
0: [2023-03-15 22:15:39,782] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_17-model_00-model_states.pt...
0: [2023-03-15 22:15:39,806] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_17-model_00-model_states.pt.
0: [2023-03-15 22:15:39,806] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_18-model_00-model_states.pt...
0: [2023-03-15 22:15:39,830] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_18-model_00-model_states.pt.
0: [2023-03-15 22:15:39,830] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_19-model_00-model_states.pt...
0: [2023-03-15 22:15:39,854] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_19-model_00-model_states.pt.
0: [2023-03-15 22:15:39,855] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_20-model_00-model_states.pt...
0: [2023-03-15 22:15:39,879] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_20-model_00-model_states.pt.
0: [2023-03-15 22:15:39,879] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/layer_22-model_00-model_states.pt...
0: [2023-03-15 22:15:39,880] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/layer_22-model_00-model_states.pt.
0: [2023-03-15 22:15:39,881] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_280m5b9400m/global_step2000/mp_rank_00_model_states.pt
0: [2023-03-15 22:15:39,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/mp_rank_00_model_states.pt...
0: [2023-03-15 22:15:39,883] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/mp_rank_00_model_states.pt.
0: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:15:39,900] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:15:39,959] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:15:39,959] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-15 22:15:39,959] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
7: [2023-03-15 22:15:39,960] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:15:39,960] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-15 22:15:39,960] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
2: [2023-03-15 22:15:39,961] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:15:39,961] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-15 22:15:39,961] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
5: [2023-03-15 22:15:39,962] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:15:39,962] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
4: [2023-03-15 22:15:39,962] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:15:39,962] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
4: [2023-03-15 22:15:39,962] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-15 22:15:39,962] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
7: [2023-03-15 22:15:39,962] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:15:39,963] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-15 22:15:39,963] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
4: [2023-03-15 22:15:39,963] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:15:39,963] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-15 22:15:39,963] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
6: [2023-03-15 22:15:39,964] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:15:39,964] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-15 22:15:39,965] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
6: [2023-03-15 22:15:39,964] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:15:39,965] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-15 22:15:39,965] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
4: [2023-03-15 22:15:39,965] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:15:39,965] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-15 22:15:39,966] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
7: [2023-03-15 22:15:39,967] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:15:39,967] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-15 22:15:39,967] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
4: [2023-03-15 22:15:39,967] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:15:39,967] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:15:39,967] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
7: [2023-03-15 22:15:39,967] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
4: [2023-03-15 22:15:39,967] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
7: [2023-03-15 22:15:39,967] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
5: [2023-03-15 22:15:39,968] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:15:39,968] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:15:39,968] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-15 22:15:39,968] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-15 22:15:39,968] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
5: [2023-03-15 22:15:39,968] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
5: [2023-03-15 22:15:39,969] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:15:39,969] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-15 22:15:39,969] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
0: [2023-03-15 22:15:39,970] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:15:39,970] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:15:39,970] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-15 22:15:39,970] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-15 22:15:39,970] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
0: [2023-03-15 22:15:39,970] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
6: [2023-03-15 22:15:39,970] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:15:39,970] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:15:39,970] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-15 22:15:39,970] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
0: [2023-03-15 22:15:39,970] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:15:39,970] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:15:39,970] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
6: [2023-03-15 22:15:39,970] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
0: [2023-03-15 22:15:39,970] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
6: [2023-03-15 22:15:39,970] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
7: [2023-03-15 22:15:39,971] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:15:39,971] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-15 22:15:39,971] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
4: [2023-03-15 22:15:39,971] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:15:39,972] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-15 22:15:39,972] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
6: [2023-03-15 22:15:39,971] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:15:39,972] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:15:39,971] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
1: [2023-03-15 22:15:39,972] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
6: [2023-03-15 22:15:39,971] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
1: [2023-03-15 22:15:39,972] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
6: [2023-03-15 22:15:39,971] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:15:39,971] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-15 22:15:39,971] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
5: [2023-03-15 22:15:39,972] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:15:39,972] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-15 22:15:39,972] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
7: [2023-03-15 22:15:39,973] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:15:39,973] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-15 22:15:39,973] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
7: [2023-03-15 22:15:39,973] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:15:39,973] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:15:39,973] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
4: [2023-03-15 22:15:39,973] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
7: [2023-03-15 22:15:39,973] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
4: [2023-03-15 22:15:39,973] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
7: [2023-03-15 22:15:39,973] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:15:39,973] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-15 22:15:39,973] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
3: [2023-03-15 22:15:39,970] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-15 22:15:39,970] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
3: [2023-03-15 22:15:39,970] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:15:39,970] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-15 22:15:39,971] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
3: [2023-03-15 22:15:39,971] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:15:39,971] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-15 22:15:39,971] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
3: [2023-03-15 22:15:39,973] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:15:39,973] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-15 22:15:39,973] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
4: [2023-03-15 22:15:39,975] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:15:39,975] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:15:39,975] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
2: [2023-03-15 22:15:39,975] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:15:39,975] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
2: [2023-03-15 22:15:39,975] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
4: [2023-03-15 22:15:39,975] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
4: [2023-03-15 22:15:39,975] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
2: [2023-03-15 22:15:39,975] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
2: [2023-03-15 22:15:39,975] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:15:39,976] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-15 22:15:39,976] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
2: [2023-03-15 22:15:39,976] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:15:39,976] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:15:39,976] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-15 22:15:39,976] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-15 22:15:39,976] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
2: [2023-03-15 22:15:39,976] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
3: [2023-03-15 22:15:39,976] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:15:39,976] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-15 22:15:39,976] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
5: [2023-03-15 22:15:39,976] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:15:39,976] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-15 22:15:39,976] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
3: [2023-03-15 22:15:39,976] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:15:39,977] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-15 22:15:39,977] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
3: [2023-03-15 22:15:39,977] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:15:39,977] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:15:39,977] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
5: [2023-03-15 22:15:39,977] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
3: [2023-03-15 22:15:39,977] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
5: [2023-03-15 22:15:39,977] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
5: [2023-03-15 22:15:39,977] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:15:39,977] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-15 22:15:39,977] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
2: [2023-03-15 22:15:39,980] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:15:39,980] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-15 22:15:39,980] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
2: [2023-03-15 22:15:39,980] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:15:39,980] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-15 22:15:39,981] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
2: [2023-03-15 22:15:39,981] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:15:39,981] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-15 22:15:39,981] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
1: [2023-03-15 22:15:39,982] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:15:39,982] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-15 22:15:39,982] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
1: [2023-03-15 22:15:39,993] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:15:39,993] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-15 22:15:39,993] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
1: [2023-03-15 22:15:39,996] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:15:39,996] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-15 22:15:39,996] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
0: [2023-03-15 22:15:40,017] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:15:40,017] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:15:40,017] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-15 22:15:40,017] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:15:40,017] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
0: [2023-03-15 22:15:40,017] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-15 22:15:40,017] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
1: [2023-03-15 22:15:40,022] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:15:40,022] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-15 22:15:40,023] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
0: [2023-03-15 22:15:40,025] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:15:40,025] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-15 22:15:40,025] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
1: [2023-03-15 22:15:40,026] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:15:40,026] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-15 22:15:40,026] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
6: [2023-03-15 22:15:40,030] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:15:40,030] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-15 22:15:40,030] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
0: [2023-03-15 22:15:40,034] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:15:40,034] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-15 22:15:40,034] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
1: [2023-03-15 22:15:40,034] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:15:40,034] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-15 22:15:40,034] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
1: [2023-03-15 22:15:40,038] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:15:40,038] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-15 22:15:40,038] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
6: [2023-03-15 22:15:40,039] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:15:40,039] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-15 22:15:40,039] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
0: [2023-03-15 22:15:40,055] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step2000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-15 22:15:40,055] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step2000 is ready now!
0:   successfully saved checkpoint at iteration    2000 to checkpoints_280m5b9400m
7: time (ms) | save-checkpoint: 744.85
7:  iteration     2010/   11269 | consumed samples:       514560 | consumed tokens:   1053818880 | elapsed time per iteration (s): 0.57 | learning rate: 1.875E-04 | global batch size:   256 | lm loss: 3.882499E+00 | grad norm: 0.332 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 450.051 | TFLOPs: 29.43 |
7:  iteration     2020/   11269 | consumed samples:       517120 | consumed tokens:   1059061760 | elapsed time per iteration (s): 0.48 | learning rate: 1.873E-04 | global batch size:   256 | lm loss: 3.884430E+00 | grad norm: 0.419 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.585 | TFLOPs: 35.22 |
7:  iteration     2030/   11269 | consumed samples:       519680 | consumed tokens:   1064304640 | elapsed time per iteration (s): 0.48 | learning rate: 1.872E-04 | global batch size:   256 | lm loss: 3.887638E+00 | grad norm: 0.405 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.488 | TFLOPs: 34.89 |
7:  iteration     2040/   11269 | consumed samples:       522240 | consumed tokens:   1069547520 | elapsed time per iteration (s): 0.48 | learning rate: 1.871E-04 | global batch size:   256 | lm loss: 3.870945E+00 | grad norm: 0.373 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.465 | TFLOPs: 35.02 |
7:  iteration     2050/   11269 | consumed samples:       524800 | consumed tokens:   1074790400 | elapsed time per iteration (s): 0.48 | learning rate: 1.869E-04 | global batch size:   256 | lm loss: 3.869390E+00 | grad norm: 0.373 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.743 | TFLOPs: 35.10 |
7:  iteration     2060/   11269 | consumed samples:       527360 | consumed tokens:   1080033280 | elapsed time per iteration (s): 0.48 | learning rate: 1.868E-04 | global batch size:   256 | lm loss: 3.876960E+00 | grad norm: 0.400 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.831 | TFLOPs: 35.04 |
7:  iteration     2070/   11269 | consumed samples:       529920 | consumed tokens:   1085276160 | elapsed time per iteration (s): 0.48 | learning rate: 1.867E-04 | global batch size:   256 | lm loss: 3.859916E+00 | grad norm: 0.336 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.341 | TFLOPs: 35.14 |
7:  iteration     2080/   11269 | consumed samples:       532480 | consumed tokens:   1090519040 | elapsed time per iteration (s): 0.48 | learning rate: 1.865E-04 | global batch size:   256 | lm loss: 3.851398E+00 | grad norm: 0.377 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 529.770 | TFLOPs: 34.65 |
7:  iteration     2090/   11269 | consumed samples:       535040 | consumed tokens:   1095761920 | elapsed time per iteration (s): 0.47 | learning rate: 1.864E-04 | global batch size:   256 | lm loss: 3.851305E+00 | grad norm: 0.338 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.674 | TFLOPs: 35.43 |
7:  iteration     2100/   11269 | consumed samples:       537600 | consumed tokens:   1101004800 | elapsed time per iteration (s): 0.48 | learning rate: 1.863E-04 | global batch size:   256 | lm loss: 3.839741E+00 | grad norm: 0.448 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.308 | TFLOPs: 35.07 |
7:  iteration     2110/   11269 | consumed samples:       540160 | consumed tokens:   1106247680 | elapsed time per iteration (s): 0.48 | learning rate: 1.861E-04 | global batch size:   256 | lm loss: 3.844390E+00 | grad norm: 0.447 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.271 | TFLOPs: 35.20 |
7:  iteration     2120/   11269 | consumed samples:       542720 | consumed tokens:   1111490560 | elapsed time per iteration (s): 0.48 | learning rate: 1.860E-04 | global batch size:   256 | lm loss: 3.840216E+00 | grad norm: 0.413 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.916 | TFLOPs: 35.18 |
7:  iteration     2130/   11269 | consumed samples:       545280 | consumed tokens:   1116733440 | elapsed time per iteration (s): 0.48 | learning rate: 1.859E-04 | global batch size:   256 | lm loss: 3.850616E+00 | grad norm: 0.372 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.176 | TFLOPs: 35.07 |
7:  iteration     2140/   11269 | consumed samples:       547840 | consumed tokens:   1121976320 | elapsed time per iteration (s): 0.47 | learning rate: 1.857E-04 | global batch size:   256 | lm loss: 3.836988E+00 | grad norm: 0.356 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.656 | TFLOPs: 35.42 |
7:  iteration     2150/   11269 | consumed samples:       550400 | consumed tokens:   1127219200 | elapsed time per iteration (s): 0.48 | learning rate: 1.856E-04 | global batch size:   256 | lm loss: 3.842624E+00 | grad norm: 0.373 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.060 | TFLOPs: 34.93 |
7:  iteration     2160/   11269 | consumed samples:       552960 | consumed tokens:   1132462080 | elapsed time per iteration (s): 0.47 | learning rate: 1.855E-04 | global batch size:   256 | lm loss: 3.845037E+00 | grad norm: 0.387 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.450 | TFLOPs: 35.28 |
7:  iteration     2170/   11269 | consumed samples:       555520 | consumed tokens:   1137704960 | elapsed time per iteration (s): 0.48 | learning rate: 1.853E-04 | global batch size:   256 | lm loss: 3.845623E+00 | grad norm: 0.396 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.978 | TFLOPs: 35.05 |
7:  iteration     2180/   11269 | consumed samples:       558080 | consumed tokens:   1142947840 | elapsed time per iteration (s): 0.48 | learning rate: 1.852E-04 | global batch size:   256 | lm loss: 3.837968E+00 | grad norm: 0.346 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.960 | TFLOPs: 35.18 |
7:  iteration     2190/   11269 | consumed samples:       560640 | consumed tokens:   1148190720 | elapsed time per iteration (s): 0.47 | learning rate: 1.850E-04 | global batch size:   256 | lm loss: 3.824232E+00 | grad norm: 0.353 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.743 | TFLOPs: 35.43 |
7:  iteration     2200/   11269 | consumed samples:       563200 | consumed tokens:   1153433600 | elapsed time per iteration (s): 0.48 | learning rate: 1.849E-04 | global batch size:   256 | lm loss: 3.828149E+00 | grad norm: 0.388 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.316 | TFLOPs: 34.88 |
7:  iteration     2210/   11269 | consumed samples:       565760 | consumed tokens:   1158676480 | elapsed time per iteration (s): 0.47 | learning rate: 1.848E-04 | global batch size:   256 | lm loss: 3.823164E+00 | grad norm: 0.339 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.684 | TFLOPs: 35.43 |
7:  iteration     2220/   11269 | consumed samples:       568320 | consumed tokens:   1163919360 | elapsed time per iteration (s): 0.48 | learning rate: 1.846E-04 | global batch size:   256 | lm loss: 3.819661E+00 | grad norm: 0.445 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 528.423 | TFLOPs: 34.56 |
7:  iteration     2230/   11269 | consumed samples:       570880 | consumed tokens:   1169162240 | elapsed time per iteration (s): 0.48 | learning rate: 1.845E-04 | global batch size:   256 | lm loss: 3.820479E+00 | grad norm: 0.373 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.924 | TFLOPs: 35.25 |
7:  iteration     2240/   11269 | consumed samples:       573440 | consumed tokens:   1174405120 | elapsed time per iteration (s): 0.47 | learning rate: 1.843E-04 | global batch size:   256 | lm loss: 3.823940E+00 | grad norm: 0.381 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.101 | TFLOPs: 35.26 |
7:  iteration     2250/   11269 | consumed samples:       576000 | consumed tokens:   1179648000 | elapsed time per iteration (s): 0.48 | learning rate: 1.842E-04 | global batch size:   256 | lm loss: 3.828367E+00 | grad norm: 0.373 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.266 | TFLOPs: 35.14 |
7:  iteration     2260/   11269 | consumed samples:       578560 | consumed tokens:   1184890880 | elapsed time per iteration (s): 0.47 | learning rate: 1.840E-04 | global batch size:   256 | lm loss: 3.815744E+00 | grad norm: 0.421 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.571 | TFLOPs: 35.29 |
7:  iteration     2270/   11269 | consumed samples:       581120 | consumed tokens:   1190133760 | elapsed time per iteration (s): 0.48 | learning rate: 1.839E-04 | global batch size:   256 | lm loss: 3.817776E+00 | grad norm: 0.358 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.299 | TFLOPs: 34.75 |
7:  iteration     2280/   11269 | consumed samples:       583680 | consumed tokens:   1195376640 | elapsed time per iteration (s): 0.47 | learning rate: 1.838E-04 | global batch size:   256 | lm loss: 3.802319E+00 | grad norm: 0.357 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.723 | TFLOPs: 35.43 |
7:  iteration     2290/   11269 | consumed samples:       586240 | consumed tokens:   1200619520 | elapsed time per iteration (s): 0.49 | learning rate: 1.836E-04 | global batch size:   256 | lm loss: 3.803706E+00 | grad norm: 0.359 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 520.957 | TFLOPs: 34.07 |
7:  iteration     2300/   11269 | consumed samples:       588800 | consumed tokens:   1205862400 | elapsed time per iteration (s): 0.47 | learning rate: 1.835E-04 | global batch size:   256 | lm loss: 3.797497E+00 | grad norm: 0.404 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.773 | TFLOPs: 35.43 |
7:  iteration     2310/   11269 | consumed samples:       591360 | consumed tokens:   1211105280 | elapsed time per iteration (s): 0.48 | learning rate: 1.833E-04 | global batch size:   256 | lm loss: 3.811961E+00 | grad norm: 0.423 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.163 | TFLOPs: 35.06 |
7:  iteration     2320/   11269 | consumed samples:       593920 | consumed tokens:   1216348160 | elapsed time per iteration (s): 0.48 | learning rate: 1.832E-04 | global batch size:   256 | lm loss: 3.815844E+00 | grad norm: 0.369 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.503 | TFLOPs: 35.22 |
7:  iteration     2330/   11269 | consumed samples:       596480 | consumed tokens:   1221591040 | elapsed time per iteration (s): 0.48 | learning rate: 1.830E-04 | global batch size:   256 | lm loss: 3.798663E+00 | grad norm: 0.354 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.136 | TFLOPs: 35.19 |
7:  iteration     2340/   11269 | consumed samples:       599040 | consumed tokens:   1226833920 | elapsed time per iteration (s): 0.47 | learning rate: 1.829E-04 | global batch size:   256 | lm loss: 3.810221E+00 | grad norm: 0.349 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.398 | TFLOPs: 35.28 |
7:  iteration     2350/   11269 | consumed samples:       601600 | consumed tokens:   1232076800 | elapsed time per iteration (s): 0.48 | learning rate: 1.827E-04 | global batch size:   256 | lm loss: 3.803239E+00 | grad norm: 0.367 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.027 | TFLOPs: 35.19 |
7:  iteration     2360/   11269 | consumed samples:       604160 | consumed tokens:   1237319680 | elapsed time per iteration (s): 0.48 | learning rate: 1.826E-04 | global batch size:   256 | lm loss: 3.800451E+00 | grad norm: 0.347 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.479 | TFLOPs: 35.22 |
7:  iteration     2370/   11269 | consumed samples:       606720 | consumed tokens:   1242562560 | elapsed time per iteration (s): 0.48 | learning rate: 1.824E-04 | global batch size:   256 | lm loss: 3.791774E+00 | grad norm: 0.419 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.122 | TFLOPs: 35.13 |
7:  iteration     2380/   11269 | consumed samples:       609280 | consumed tokens:   1247805440 | elapsed time per iteration (s): 0.47 | learning rate: 1.823E-04 | global batch size:   256 | lm loss: 3.802037E+00 | grad norm: 0.418 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.485 | TFLOPs: 35.41 |
7:  iteration     2390/   11269 | consumed samples:       611840 | consumed tokens:   1253048320 | elapsed time per iteration (s): 0.47 | learning rate: 1.821E-04 | global batch size:   256 | lm loss: 3.794889E+00 | grad norm: 0.400 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.245 | TFLOPs: 35.27 |
7:  iteration     2400/   11269 | consumed samples:       614400 | consumed tokens:   1258291200 | elapsed time per iteration (s): 0.47 | learning rate: 1.820E-04 | global batch size:   256 | lm loss: 3.777916E+00 | grad norm: 0.348 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.599 | TFLOPs: 35.42 |
7:  iteration     2410/   11269 | consumed samples:       616960 | consumed tokens:   1263534080 | elapsed time per iteration (s): 0.47 | learning rate: 1.818E-04 | global batch size:   256 | lm loss: 3.786474E+00 | grad norm: 0.390 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.462 | TFLOPs: 35.41 |
7:  iteration     2420/   11269 | consumed samples:       619520 | consumed tokens:   1268776960 | elapsed time per iteration (s): 0.48 | learning rate: 1.817E-04 | global batch size:   256 | lm loss: 3.784957E+00 | grad norm: 0.391 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.828 | TFLOPs: 34.98 |
7:  iteration     2430/   11269 | consumed samples:       622080 | consumed tokens:   1274019840 | elapsed time per iteration (s): 0.47 | learning rate: 1.815E-04 | global batch size:   256 | lm loss: 3.775450E+00 | grad norm: 0.358 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.556 | TFLOPs: 35.42 |
7:  iteration     2440/   11269 | consumed samples:       624640 | consumed tokens:   1279262720 | elapsed time per iteration (s): 0.48 | learning rate: 1.814E-04 | global batch size:   256 | lm loss: 3.775534E+00 | grad norm: 0.392 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.391 | TFLOPs: 35.21 |
7:  iteration     2450/   11269 | consumed samples:       627200 | consumed tokens:   1284505600 | elapsed time per iteration (s): 0.47 | learning rate: 1.812E-04 | global batch size:   256 | lm loss: 3.786253E+00 | grad norm: 0.368 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.987 | TFLOPs: 35.25 |
7:  iteration     2460/   11269 | consumed samples:       629760 | consumed tokens:   1289748480 | elapsed time per iteration (s): 0.47 | learning rate: 1.810E-04 | global batch size:   256 | lm loss: 3.775782E+00 | grad norm: 0.347 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.429 | TFLOPs: 35.28 |
7:  iteration     2470/   11269 | consumed samples:       632320 | consumed tokens:   1294991360 | elapsed time per iteration (s): 0.47 | learning rate: 1.809E-04 | global batch size:   256 | lm loss: 3.776648E+00 | grad norm: 0.375 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.418 | TFLOPs: 35.28 |
7:  iteration     2480/   11269 | consumed samples:       634880 | consumed tokens:   1300234240 | elapsed time per iteration (s): 0.47 | learning rate: 1.807E-04 | global batch size:   256 | lm loss: 3.790529E+00 | grad norm: 0.368 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.532 | TFLOPs: 35.42 |
7:  iteration     2490/   11269 | consumed samples:       637440 | consumed tokens:   1305477120 | elapsed time per iteration (s): 0.47 | learning rate: 1.806E-04 | global batch size:   256 | lm loss: 3.769222E+00 | grad norm: 0.400 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.451 | TFLOPs: 35.41 |
7:  iteration     2500/   11269 | consumed samples:       640000 | consumed tokens:   1310720000 | elapsed time per iteration (s): 0.47 | learning rate: 1.804E-04 | global batch size:   256 | lm loss: 3.775280E+00 | grad norm: 0.390 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.491 | TFLOPs: 35.41 |
7:  iteration     2510/   11269 | consumed samples:       642560 | consumed tokens:   1315962880 | elapsed time per iteration (s): 0.48 | learning rate: 1.803E-04 | global batch size:   256 | lm loss: 3.749633E+00 | grad norm: 0.373 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.124 | TFLOPs: 35.19 |
7:  iteration     2520/   11269 | consumed samples:       645120 | consumed tokens:   1321205760 | elapsed time per iteration (s): 0.47 | learning rate: 1.801E-04 | global batch size:   256 | lm loss: 3.765163E+00 | grad norm: 0.363 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.805 | TFLOPs: 35.43 |
7:  iteration     2530/   11269 | consumed samples:       647680 | consumed tokens:   1326448640 | elapsed time per iteration (s): 0.47 | learning rate: 1.799E-04 | global batch size:   256 | lm loss: 3.776926E+00 | grad norm: 0.383 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.633 | TFLOPs: 35.29 |
7:  iteration     2540/   11269 | consumed samples:       650240 | consumed tokens:   1331691520 | elapsed time per iteration (s): 0.47 | learning rate: 1.798E-04 | global batch size:   256 | lm loss: 3.774195E+00 | grad norm: 0.371 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.261 | TFLOPs: 35.33 |
7:  iteration     2550/   11269 | consumed samples:       652800 | consumed tokens:   1336934400 | elapsed time per iteration (s): 0.48 | learning rate: 1.796E-04 | global batch size:   256 | lm loss: 3.748422E+00 | grad norm: 0.351 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.331 | TFLOPs: 35.14 |
7:  iteration     2560/   11269 | consumed samples:       655360 | consumed tokens:   1342177280 | elapsed time per iteration (s): 0.47 | learning rate: 1.795E-04 | global batch size:   256 | lm loss: 3.760729E+00 | grad norm: 0.373 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.308 | TFLOPs: 35.40 |
7:  iteration     2570/   11269 | consumed samples:       657920 | consumed tokens:   1347420160 | elapsed time per iteration (s): 0.48 | learning rate: 1.793E-04 | global batch size:   256 | lm loss: 3.756708E+00 | grad norm: 0.375 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.969 | TFLOPs: 34.99 |
7:  iteration     2580/   11269 | consumed samples:       660480 | consumed tokens:   1352663040 | elapsed time per iteration (s): 0.48 | learning rate: 1.791E-04 | global batch size:   256 | lm loss: 3.748076E+00 | grad norm: 0.386 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.272 | TFLOPs: 35.14 |
7:  iteration     2590/   11269 | consumed samples:       663040 | consumed tokens:   1357905920 | elapsed time per iteration (s): 0.48 | learning rate: 1.790E-04 | global batch size:   256 | lm loss: 3.750365E+00 | grad norm: 0.349 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.705 | TFLOPs: 35.17 |
7:  iteration     2600/   11269 | consumed samples:       665600 | consumed tokens:   1363148800 | elapsed time per iteration (s): 0.47 | learning rate: 1.788E-04 | global batch size:   256 | lm loss: 3.749382E+00 | grad norm: 0.398 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.449 | TFLOPs: 35.41 |
7:  iteration     2610/   11269 | consumed samples:       668160 | consumed tokens:   1368391680 | elapsed time per iteration (s): 0.48 | learning rate: 1.786E-04 | global batch size:   256 | lm loss: 3.756418E+00 | grad norm: 0.353 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.458 | TFLOPs: 35.08 |
7:  iteration     2620/   11269 | consumed samples:       670720 | consumed tokens:   1373634560 | elapsed time per iteration (s): 0.48 | learning rate: 1.785E-04 | global batch size:   256 | lm loss: 3.745824E+00 | grad norm: 0.409 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.754 | TFLOPs: 35.23 |
7:  iteration     2630/   11269 | consumed samples:       673280 | consumed tokens:   1378877440 | elapsed time per iteration (s): 0.48 | learning rate: 1.783E-04 | global batch size:   256 | lm loss: 3.745185E+00 | grad norm: 0.359 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.268 | TFLOPs: 35.14 |
7:  iteration     2640/   11269 | consumed samples:       675840 | consumed tokens:   1384120320 | elapsed time per iteration (s): 0.48 | learning rate: 1.782E-04 | global batch size:   256 | lm loss: 3.752087E+00 | grad norm: 0.375 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.026 | TFLOPs: 35.19 |
7:  iteration     2650/   11269 | consumed samples:       678400 | consumed tokens:   1389363200 | elapsed time per iteration (s): 0.48 | learning rate: 1.780E-04 | global batch size:   256 | lm loss: 3.737976E+00 | grad norm: 0.350 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.916 | TFLOPs: 35.11 |
7:  iteration     2660/   11269 | consumed samples:       680960 | consumed tokens:   1394606080 | elapsed time per iteration (s): 0.48 | learning rate: 1.778E-04 | global batch size:   256 | lm loss: 3.742223E+00 | grad norm: 0.337 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.095 | TFLOPs: 35.06 |
7:  iteration     2670/   11269 | consumed samples:       683520 | consumed tokens:   1399848960 | elapsed time per iteration (s): 0.48 | learning rate: 1.777E-04 | global batch size:   256 | lm loss: 3.751134E+00 | grad norm: 0.438 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 532.983 | TFLOPs: 34.86 |
7:  iteration     2680/   11269 | consumed samples:       686080 | consumed tokens:   1405091840 | elapsed time per iteration (s): 0.48 | learning rate: 1.775E-04 | global batch size:   256 | lm loss: 3.736777E+00 | grad norm: 0.458 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.631 | TFLOPs: 35.03 |
7:  iteration     2690/   11269 | consumed samples:       688640 | consumed tokens:   1410334720 | elapsed time per iteration (s): 0.47 | learning rate: 1.773E-04 | global batch size:   256 | lm loss: 3.746079E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.593 | TFLOPs: 35.35 |
7:  iteration     2700/   11269 | consumed samples:       691200 | consumed tokens:   1415577600 | elapsed time per iteration (s): 0.48 | learning rate: 1.772E-04 | global batch size:   256 | lm loss: 3.735019E+00 | grad norm: 0.388 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.296 | TFLOPs: 35.14 |
7:  iteration     2710/   11269 | consumed samples:       693760 | consumed tokens:   1420820480 | elapsed time per iteration (s): 0.47 | learning rate: 1.770E-04 | global batch size:   256 | lm loss: 3.735714E+00 | grad norm: 0.377 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.660 | TFLOPs: 35.42 |
7:  iteration     2720/   11269 | consumed samples:       696320 | consumed tokens:   1426063360 | elapsed time per iteration (s): 0.48 | learning rate: 1.768E-04 | global batch size:   256 | lm loss: 3.720799E+00 | grad norm: 0.411 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.175 | TFLOPs: 35.13 |
7:  iteration     2730/   11269 | consumed samples:       698880 | consumed tokens:   1431306240 | elapsed time per iteration (s): 0.48 | learning rate: 1.766E-04 | global batch size:   256 | lm loss: 3.731408E+00 | grad norm: 0.389 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.472 | TFLOPs: 35.22 |
7:  iteration     2740/   11269 | consumed samples:       701440 | consumed tokens:   1436549120 | elapsed time per iteration (s): 0.47 | learning rate: 1.765E-04 | global batch size:   256 | lm loss: 3.739030E+00 | grad norm: 0.408 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.521 | TFLOPs: 35.42 |
7:  iteration     2750/   11269 | consumed samples:       704000 | consumed tokens:   1441792000 | elapsed time per iteration (s): 0.48 | learning rate: 1.763E-04 | global batch size:   256 | lm loss: 3.731999E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.702 | TFLOPs: 35.03 |
7:  iteration     2760/   11269 | consumed samples:       706560 | consumed tokens:   1447034880 | elapsed time per iteration (s): 0.47 | learning rate: 1.761E-04 | global batch size:   256 | lm loss: 3.724368E+00 | grad norm: 0.338 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.640 | TFLOPs: 35.42 |
7:  iteration     2770/   11269 | consumed samples:       709120 | consumed tokens:   1452277760 | elapsed time per iteration (s): 0.48 | learning rate: 1.760E-04 | global batch size:   256 | lm loss: 3.731783E+00 | grad norm: 0.339 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.201 | TFLOPs: 35.07 |
7:  iteration     2780/   11269 | consumed samples:       711680 | consumed tokens:   1457520640 | elapsed time per iteration (s): 0.48 | learning rate: 1.758E-04 | global batch size:   256 | lm loss: 3.711679E+00 | grad norm: 0.355 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.784 | TFLOPs: 35.11 |
7:  iteration     2790/   11269 | consumed samples:       714240 | consumed tokens:   1462763520 | elapsed time per iteration (s): 0.47 | learning rate: 1.756E-04 | global batch size:   256 | lm loss: 3.721898E+00 | grad norm: 0.349 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.990 | TFLOPs: 35.45 |
7:  iteration     2800/   11269 | consumed samples:       716800 | consumed tokens:   1468006400 | elapsed time per iteration (s): 0.48 | learning rate: 1.754E-04 | global batch size:   256 | lm loss: 3.718946E+00 | grad norm: 0.340 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.908 | TFLOPs: 35.11 |
7:  iteration     2810/   11269 | consumed samples:       719360 | consumed tokens:   1473249280 | elapsed time per iteration (s): 0.47 | learning rate: 1.753E-04 | global batch size:   256 | lm loss: 3.719162E+00 | grad norm: 0.342 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.913 | TFLOPs: 35.44 |
7:  iteration     2820/   11269 | consumed samples:       721920 | consumed tokens:   1478492160 | elapsed time per iteration (s): 0.48 | learning rate: 1.751E-04 | global batch size:   256 | lm loss: 3.706026E+00 | grad norm: 0.358 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.900 | TFLOPs: 35.11 |
7:  iteration     2830/   11269 | consumed samples:       724480 | consumed tokens:   1483735040 | elapsed time per iteration (s): 0.47 | learning rate: 1.749E-04 | global batch size:   256 | lm loss: 3.696422E+00 | grad norm: 0.388 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.468 | TFLOPs: 35.41 |
7:  iteration     2840/   11269 | consumed samples:       727040 | consumed tokens:   1488977920 | elapsed time per iteration (s): 0.47 | learning rate: 1.747E-04 | global batch size:   256 | lm loss: 3.692456E+00 | grad norm: 0.378 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.642 | TFLOPs: 35.36 |
7:  iteration     2850/   11269 | consumed samples:       729600 | consumed tokens:   1494220800 | elapsed time per iteration (s): 0.47 | learning rate: 1.746E-04 | global batch size:   256 | lm loss: 3.685821E+00 | grad norm: 0.368 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.673 | TFLOPs: 35.43 |
7:  iteration     2860/   11269 | consumed samples:       732160 | consumed tokens:   1499463680 | elapsed time per iteration (s): 0.47 | learning rate: 1.744E-04 | global batch size:   256 | lm loss: 3.705552E+00 | grad norm: 0.393 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.618 | TFLOPs: 35.29 |
7:  iteration     2870/   11269 | consumed samples:       734720 | consumed tokens:   1504706560 | elapsed time per iteration (s): 0.48 | learning rate: 1.742E-04 | global batch size:   256 | lm loss: 3.698108E+00 | grad norm: 0.348 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 532.991 | TFLOPs: 34.86 |
7:  iteration     2880/   11269 | consumed samples:       737280 | consumed tokens:   1509949440 | elapsed time per iteration (s): 0.49 | learning rate: 1.740E-04 | global batch size:   256 | lm loss: 3.706473E+00 | grad norm: 0.389 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 519.494 | TFLOPs: 33.97 |
7:  iteration     2890/   11269 | consumed samples:       739840 | consumed tokens:   1515192320 | elapsed time per iteration (s): 0.47 | learning rate: 1.739E-04 | global batch size:   256 | lm loss: 3.711499E+00 | grad norm: 0.367 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.080 | TFLOPs: 35.45 |
7:  iteration     2900/   11269 | consumed samples:       742400 | consumed tokens:   1520435200 | elapsed time per iteration (s): 0.48 | learning rate: 1.737E-04 | global batch size:   256 | lm loss: 3.710240E+00 | grad norm: 0.362 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.097 | TFLOPs: 35.19 |
7:  iteration     2910/   11269 | consumed samples:       744960 | consumed tokens:   1525678080 | elapsed time per iteration (s): 0.47 | learning rate: 1.735E-04 | global batch size:   256 | lm loss: 3.713371E+00 | grad norm: 0.327 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.133 | TFLOPs: 35.46 |
7:  iteration     2920/   11269 | consumed samples:       747520 | consumed tokens:   1530920960 | elapsed time per iteration (s): 0.48 | learning rate: 1.733E-04 | global batch size:   256 | lm loss: 3.702215E+00 | grad norm: 0.340 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.932 | TFLOPs: 35.25 |
7:  iteration     2930/   11269 | consumed samples:       750080 | consumed tokens:   1536163840 | elapsed time per iteration (s): 0.47 | learning rate: 1.731E-04 | global batch size:   256 | lm loss: 3.699169E+00 | grad norm: 0.333 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.240 | TFLOPs: 35.27 |
7:  iteration     2940/   11269 | consumed samples:       752640 | consumed tokens:   1541406720 | elapsed time per iteration (s): 0.47 | learning rate: 1.730E-04 | global batch size:   256 | lm loss: 3.703460E+00 | grad norm: 0.393 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.050 | TFLOPs: 35.45 |
7:  iteration     2950/   11269 | consumed samples:       755200 | consumed tokens:   1546649600 | elapsed time per iteration (s): 0.48 | learning rate: 1.728E-04 | global batch size:   256 | lm loss: 3.700461E+00 | grad norm: 0.430 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.859 | TFLOPs: 35.04 |
7:  iteration     2960/   11269 | consumed samples:       757760 | consumed tokens:   1551892480 | elapsed time per iteration (s): 0.47 | learning rate: 1.726E-04 | global batch size:   256 | lm loss: 3.681445E+00 | grad norm: 0.329 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.029 | TFLOPs: 35.38 |
7:  iteration     2970/   11269 | consumed samples:       760320 | consumed tokens:   1557135360 | elapsed time per iteration (s): 0.47 | learning rate: 1.724E-04 | global batch size:   256 | lm loss: 3.703711E+00 | grad norm: 0.329 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.918 | TFLOPs: 35.44 |
7:  iteration     2980/   11269 | consumed samples:       762880 | consumed tokens:   1562378240 | elapsed time per iteration (s): 0.48 | learning rate: 1.722E-04 | global batch size:   256 | lm loss: 3.691098E+00 | grad norm: 0.392 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.650 | TFLOPs: 35.03 |
7:  iteration     2990/   11269 | consumed samples:       765440 | consumed tokens:   1567621120 | elapsed time per iteration (s): 0.47 | learning rate: 1.720E-04 | global batch size:   256 | lm loss: 3.701260E+00 | grad norm: 0.340 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.292 | TFLOPs: 35.40 |
7:  iteration     3000/   11269 | consumed samples:       768000 | consumed tokens:   1572864000 | elapsed time per iteration (s): 0.48 | learning rate: 1.719E-04 | global batch size:   256 | lm loss: 3.698249E+00 | grad norm: 0.342 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 532.464 | TFLOPs: 34.82 |
7: -----------------------------------------------------------------------------------------------
7: validation loss at iteration 3000 | lm loss value: 3.670590E+00 | lm loss PPL: 3.927506E+01 | 
7: -----------------------------------------------------------------------------------------------
0: saving checkpoint at iteration    3000 to checkpoints_280m5b9400m
0: [2023-03-15 22:23:36,091] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step3000 is begin to save!
0: [2023-03-15 22:23:36,094] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_01-model_00-model_states.pt...
0: [2023-03-15 22:23:36,207] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_01-model_00-model_states.pt.
0: [2023-03-15 22:23:36,207] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_03-model_00-model_states.pt...
0: [2023-03-15 22:23:36,232] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_03-model_00-model_states.pt.
0: [2023-03-15 22:23:36,232] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_04-model_00-model_states.pt...
0: [2023-03-15 22:23:36,256] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_04-model_00-model_states.pt.
0: [2023-03-15 22:23:36,256] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_05-model_00-model_states.pt...
0: [2023-03-15 22:23:36,280] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_05-model_00-model_states.pt.
0: [2023-03-15 22:23:36,280] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_06-model_00-model_states.pt...
0: [2023-03-15 22:23:36,304] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_06-model_00-model_states.pt.
0: [2023-03-15 22:23:36,304] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_07-model_00-model_states.pt...
0: [2023-03-15 22:23:36,328] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_07-model_00-model_states.pt.
0: [2023-03-15 22:23:36,328] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_08-model_00-model_states.pt...
0: [2023-03-15 22:23:36,352] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_08-model_00-model_states.pt.
0: [2023-03-15 22:23:36,352] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_09-model_00-model_states.pt...
0: [2023-03-15 22:23:36,376] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_09-model_00-model_states.pt.
0: [2023-03-15 22:23:36,377] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_10-model_00-model_states.pt...
0: [2023-03-15 22:23:36,400] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_10-model_00-model_states.pt.
0: [2023-03-15 22:23:36,400] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_11-model_00-model_states.pt...
0: [2023-03-15 22:23:36,424] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_11-model_00-model_states.pt.
0: [2023-03-15 22:23:36,424] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_12-model_00-model_states.pt...
0: [2023-03-15 22:23:36,448] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_12-model_00-model_states.pt.
0: [2023-03-15 22:23:36,448] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_13-model_00-model_states.pt...
0: [2023-03-15 22:23:36,472] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_13-model_00-model_states.pt.
0: [2023-03-15 22:23:36,472] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_14-model_00-model_states.pt...
0: [2023-03-15 22:23:36,496] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_14-model_00-model_states.pt.
0: [2023-03-15 22:23:36,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_15-model_00-model_states.pt...
0: [2023-03-15 22:23:36,519] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_15-model_00-model_states.pt.
0: [2023-03-15 22:23:36,520] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_16-model_00-model_states.pt...
0: [2023-03-15 22:23:36,543] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_16-model_00-model_states.pt.
0: [2023-03-15 22:23:36,543] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_17-model_00-model_states.pt...
0: [2023-03-15 22:23:36,567] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_17-model_00-model_states.pt.
0: [2023-03-15 22:23:36,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_18-model_00-model_states.pt...
0: [2023-03-15 22:23:36,591] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_18-model_00-model_states.pt.
0: [2023-03-15 22:23:36,591] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_19-model_00-model_states.pt...
0: [2023-03-15 22:23:36,615] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_19-model_00-model_states.pt.
0: [2023-03-15 22:23:36,615] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_20-model_00-model_states.pt...
0: [2023-03-15 22:23:36,639] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_20-model_00-model_states.pt.
0: [2023-03-15 22:23:36,639] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/layer_22-model_00-model_states.pt...
0: [2023-03-15 22:23:36,640] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/layer_22-model_00-model_states.pt.
0: [2023-03-15 22:23:36,641] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_280m5b9400m/global_step3000/mp_rank_00_model_states.pt
0: [2023-03-15 22:23:36,641] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/mp_rank_00_model_states.pt...
0: [2023-03-15 22:23:36,643] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/mp_rank_00_model_states.pt.
0: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:23:36,660] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:23:36,719] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:23:36,719] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:23:36,719] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:23:36,719] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-15 22:23:36,719] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-15 22:23:36,719] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
2: [2023-03-15 22:23:36,719] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:23:36,719] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
2: [2023-03-15 22:23:36,720] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-15 22:23:36,720] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
0: [2023-03-15 22:23:36,720] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:23:36,720] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:23:36,720] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-15 22:23:36,720] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
7: [2023-03-15 22:23:36,720] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:23:36,720] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-15 22:23:36,721] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
6: [2023-03-15 22:23:36,722] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:23:36,722] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-15 22:23:36,722] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
2: [2023-03-15 22:23:36,723] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:23:36,723] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-15 22:23:36,724] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
6: [2023-03-15 22:23:36,724] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:23:36,724] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
4: [2023-03-15 22:23:36,724] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:23:36,724] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
4: [2023-03-15 22:23:36,724] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:23:36,724] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-15 22:23:36,724] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
4: [2023-03-15 22:23:36,724] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-15 22:23:36,724] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
4: [2023-03-15 22:23:36,724] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:23:36,724] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-15 22:23:36,724] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
2: [2023-03-15 22:23:36,724] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:23:36,725] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-15 22:23:36,725] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
6: [2023-03-15 22:23:36,725] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:23:36,725] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-15 22:23:36,725] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
1: [2023-03-15 22:23:36,725] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:23:36,725] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-15 22:23:36,725] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
4: [2023-03-15 22:23:36,726] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:23:36,726] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-15 22:23:36,726] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
4: [2023-03-15 22:23:36,726] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:23:36,726] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-15 22:23:36,726] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
6: [2023-03-15 22:23:36,727] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:23:36,727] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-15 22:23:36,728] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
5: [2023-03-15 22:23:36,728] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:23:36,728] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:23:36,728] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:23:36,728] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-15 22:23:36,728] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-15 22:23:36,728] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
5: [2023-03-15 22:23:36,728] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-15 22:23:36,728] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
5: [2023-03-15 22:23:36,728] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
7: [2023-03-15 22:23:36,728] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:23:36,729] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-15 22:23:36,729] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
7: [2023-03-15 22:23:36,729] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:23:36,729] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:23:36,729] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-15 22:23:36,729] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-15 22:23:36,729] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
7: [2023-03-15 22:23:36,729] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
4: [2023-03-15 22:23:36,729] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:23:36,729] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-15 22:23:36,729] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
7: [2023-03-15 22:23:36,729] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:23:36,729] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-15 22:23:36,729] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
0: [2023-03-15 22:23:36,729] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:23:36,729] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:23:36,730] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-15 22:23:36,730] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-15 22:23:36,730] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
0: [2023-03-15 22:23:36,730] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
1: [2023-03-15 22:23:36,731] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:23:36,731] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-15 22:23:36,731] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
2: [2023-03-15 22:23:36,726] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:23:36,726] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-15 22:23:36,726] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
2: [2023-03-15 22:23:36,726] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:23:36,726] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-15 22:23:36,726] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
2: [2023-03-15 22:23:36,728] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:23:36,728] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-15 22:23:36,728] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
2: [2023-03-15 22:23:36,731] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:23:36,731] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-15 22:23:36,731] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
2: [2023-03-15 22:23:36,733] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:23:36,733] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-15 22:23:36,733] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
6: [2023-03-15 22:23:36,735] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:23:36,735] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-15 22:23:36,736] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
4: [2023-03-15 22:23:36,736] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:23:36,736] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-15 22:23:36,736] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
0: [2023-03-15 22:23:36,741] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:23:36,741] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:23:36,741] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:23:36,741] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-15 22:23:36,741] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-15 22:23:36,741] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-15 22:23:36,741] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
0: [2023-03-15 22:23:36,741] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
0: [2023-03-15 22:23:36,741] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
5: [2023-03-15 22:23:36,741] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:23:36,719] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-15 22:23:36,719] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
3: [2023-03-15 22:23:36,721] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:23:36,722] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-15 22:23:36,722] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
3: [2023-03-15 22:23:36,725] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:23:36,725] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:23:36,725] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-15 22:23:36,725] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-15 22:23:36,725] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
3: [2023-03-15 22:23:36,725] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
3: [2023-03-15 22:23:36,727] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:23:36,727] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-15 22:23:36,727] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
3: [2023-03-15 22:23:36,732] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:23:36,732] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:23:36,732] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-15 22:23:36,732] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-15 22:23:36,732] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
3: [2023-03-15 22:23:36,732] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
3: [2023-03-15 22:23:36,732] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:23:36,732] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-15 22:23:36,732] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
5: [2023-03-15 22:23:36,741] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-15 22:23:36,741] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
5: [2023-03-15 22:23:36,741] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:23:36,741] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:23:36,742] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-15 22:23:36,742] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-15 22:23:36,742] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
5: [2023-03-15 22:23:36,742] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
4: [2023-03-15 22:23:36,760] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:23:36,760] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-15 22:23:36,760] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
0: [2023-03-15 22:23:36,762] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:23:36,762] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-15 22:23:36,763] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
1: [2023-03-15 22:23:36,773] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:23:36,773] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-15 22:23:36,773] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
7: [2023-03-15 22:23:36,781] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:23:36,781] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-15 22:23:36,781] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
6: [2023-03-15 22:23:36,783] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:23:36,783] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-15 22:23:36,784] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
0: [2023-03-15 22:23:36,786] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-15 22:23:36,786] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
1: [2023-03-15 22:23:36,789] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:23:36,789] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-15 22:23:36,789] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:23:36,789] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
1: [2023-03-15 22:23:36,789] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-15 22:23:36,789] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
1: [2023-03-15 22:23:36,793] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:23:36,793] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-15 22:23:36,793] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
1: [2023-03-15 22:23:36,796] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:23:36,796] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-15 22:23:36,796] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
1: [2023-03-15 22:23:36,803] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:23:36,803] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-15 22:23:36,803] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
5: [2023-03-15 22:23:36,810] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:23:36,810] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-15 22:23:36,810] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
7: [2023-03-15 22:23:36,812] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:23:36,812] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-15 22:23:36,812] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
7: [2023-03-15 22:23:36,813] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:23:36,813] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-15 22:23:36,813] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
5: [2023-03-15 22:23:36,818] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:23:36,819] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step3000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-15 22:23:36,819] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step3000 is ready now!
0:   successfully saved checkpoint at iteration    3000 to checkpoints_280m5b9400m
7: time (ms) | save-checkpoint: 732.02
7:  iteration     3010/   11269 | consumed samples:       770560 | consumed tokens:   1578106880 | elapsed time per iteration (s): 0.56 | learning rate: 1.717E-04 | global batch size:   256 | lm loss: 3.683694E+00 | grad norm: 0.339 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 457.997 | TFLOPs: 29.95 |
7:  iteration     3020/   11269 | consumed samples:       773120 | consumed tokens:   1583349760 | elapsed time per iteration (s): 0.48 | learning rate: 1.715E-04 | global batch size:   256 | lm loss: 3.698830E+00 | grad norm: 0.354 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.913 | TFLOPs: 34.98 |
7:  iteration     3030/   11269 | consumed samples:       775680 | consumed tokens:   1588592640 | elapsed time per iteration (s): 0.48 | learning rate: 1.713E-04 | global batch size:   256 | lm loss: 3.693024E+00 | grad norm: 0.355 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.764 | TFLOPs: 35.23 |
7:  iteration     3040/   11269 | consumed samples:       778240 | consumed tokens:   1593835520 | elapsed time per iteration (s): 0.47 | learning rate: 1.711E-04 | global batch size:   256 | lm loss: 3.683218E+00 | grad norm: 0.348 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.203 | TFLOPs: 35.46 |
7:  iteration     3050/   11269 | consumed samples:       780800 | consumed tokens:   1599078400 | elapsed time per iteration (s): 0.47 | learning rate: 1.709E-04 | global batch size:   256 | lm loss: 3.676220E+00 | grad norm: 0.401 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.036 | TFLOPs: 35.45 |
7:  iteration     3060/   11269 | consumed samples:       783360 | consumed tokens:   1604321280 | elapsed time per iteration (s): 0.47 | learning rate: 1.707E-04 | global batch size:   256 | lm loss: 3.687749E+00 | grad norm: 0.366 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.487 | TFLOPs: 35.35 |
7:  iteration     3070/   11269 | consumed samples:       785920 | consumed tokens:   1609564160 | elapsed time per iteration (s): 0.48 | learning rate: 1.706E-04 | global batch size:   256 | lm loss: 3.673960E+00 | grad norm: 0.358 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.376 | TFLOPs: 35.21 |
7:  iteration     3080/   11269 | consumed samples:       788480 | consumed tokens:   1614807040 | elapsed time per iteration (s): 0.48 | learning rate: 1.704E-04 | global batch size:   256 | lm loss: 3.676728E+00 | grad norm: 0.352 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.347 | TFLOPs: 35.14 |
7:  iteration     3090/   11269 | consumed samples:       791040 | consumed tokens:   1620049920 | elapsed time per iteration (s): 0.48 | learning rate: 1.702E-04 | global batch size:   256 | lm loss: 3.670569E+00 | grad norm: 0.326 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.855 | TFLOPs: 35.11 |
7:  iteration     3100/   11269 | consumed samples:       793600 | consumed tokens:   1625292800 | elapsed time per iteration (s): 0.48 | learning rate: 1.700E-04 | global batch size:   256 | lm loss: 3.669370E+00 | grad norm: 0.374 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.138 | TFLOPs: 35.13 |
7:  iteration     3110/   11269 | consumed samples:       796160 | consumed tokens:   1630535680 | elapsed time per iteration (s): 0.48 | learning rate: 1.698E-04 | global batch size:   256 | lm loss: 3.682632E+00 | grad norm: 0.365 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.382 | TFLOPs: 35.01 |
7:  iteration     3120/   11269 | consumed samples:       798720 | consumed tokens:   1635778560 | elapsed time per iteration (s): 0.47 | learning rate: 1.696E-04 | global batch size:   256 | lm loss: 3.686160E+00 | grad norm: 0.436 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.758 | TFLOPs: 35.43 |
7:  iteration     3130/   11269 | consumed samples:       801280 | consumed tokens:   1641021440 | elapsed time per iteration (s): 0.48 | learning rate: 1.694E-04 | global batch size:   256 | lm loss: 3.668792E+00 | grad norm: 0.359 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.501 | TFLOPs: 35.09 |
7:  iteration     3140/   11269 | consumed samples:       803840 | consumed tokens:   1646264320 | elapsed time per iteration (s): 0.47 | learning rate: 1.692E-04 | global batch size:   256 | lm loss: 3.671898E+00 | grad norm: 0.333 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.615 | TFLOPs: 35.49 |
7:  iteration     3150/   11269 | consumed samples:       806400 | consumed tokens:   1651507200 | elapsed time per iteration (s): 0.47 | learning rate: 1.690E-04 | global batch size:   256 | lm loss: 3.662459E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.183 | TFLOPs: 35.26 |
7:  iteration     3160/   11269 | consumed samples:       808960 | consumed tokens:   1656750080 | elapsed time per iteration (s): 0.47 | learning rate: 1.688E-04 | global batch size:   256 | lm loss: 3.664626E+00 | grad norm: 0.341 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.501 | TFLOPs: 35.48 |
7:  iteration     3170/   11269 | consumed samples:       811520 | consumed tokens:   1661992960 | elapsed time per iteration (s): 0.47 | learning rate: 1.687E-04 | global batch size:   256 | lm loss: 3.653599E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.668 | TFLOPs: 35.29 |
7:  iteration     3180/   11269 | consumed samples:       814080 | consumed tokens:   1667235840 | elapsed time per iteration (s): 0.47 | learning rate: 1.685E-04 | global batch size:   256 | lm loss: 3.661003E+00 | grad norm: 0.326 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.677 | TFLOPs: 35.49 |
7:  iteration     3190/   11269 | consumed samples:       816640 | consumed tokens:   1672478720 | elapsed time per iteration (s): 0.48 | learning rate: 1.683E-04 | global batch size:   256 | lm loss: 3.694192E+00 | grad norm: 4.730 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.714 | TFLOPs: 35.10 |
7:  iteration     3200/   11269 | consumed samples:       819200 | consumed tokens:   1677721600 | elapsed time per iteration (s): 0.47 | learning rate: 1.681E-04 | global batch size:   256 | lm loss: 3.942126E+00 | grad norm: 3.855 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.183 | TFLOPs: 35.33 |
7:  iteration     3210/   11269 | consumed samples:       821760 | consumed tokens:   1682964480 | elapsed time per iteration (s): 0.48 | learning rate: 1.679E-04 | global batch size:   256 | lm loss: 3.889553E+00 | grad norm: 1.606 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.353 | TFLOPs: 34.75 |
7:  iteration     3220/   11269 | consumed samples:       824320 | consumed tokens:   1688207360 | elapsed time per iteration (s): 0.47 | learning rate: 1.677E-04 | global batch size:   256 | lm loss: 3.836391E+00 | grad norm: 0.767 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.287 | TFLOPs: 35.40 |
7:  iteration     3230/   11269 | consumed samples:       826880 | consumed tokens:   1693450240 | elapsed time per iteration (s): 0.48 | learning rate: 1.675E-04 | global batch size:   256 | lm loss: 3.749614E+00 | grad norm: 0.376 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.586 | TFLOPs: 35.22 |
7:  iteration     3240/   11269 | consumed samples:       829440 | consumed tokens:   1698693120 | elapsed time per iteration (s): 0.47 | learning rate: 1.673E-04 | global batch size:   256 | lm loss: 3.722608E+00 | grad norm: 0.400 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.081 | TFLOPs: 35.45 |
7:  iteration     3250/   11269 | consumed samples:       832000 | consumed tokens:   1703936000 | elapsed time per iteration (s): 0.47 | learning rate: 1.671E-04 | global batch size:   256 | lm loss: 3.688021E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.870 | TFLOPs: 35.44 |
7:  iteration     3260/   11269 | consumed samples:       834560 | consumed tokens:   1709178880 | elapsed time per iteration (s): 0.47 | learning rate: 1.669E-04 | global batch size:   256 | lm loss: 3.684510E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.186 | TFLOPs: 35.46 |
7:  iteration     3270/   11269 | consumed samples:       837120 | consumed tokens:   1714421760 | elapsed time per iteration (s): 0.48 | learning rate: 1.667E-04 | global batch size:   256 | lm loss: 3.676788E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.360 | TFLOPs: 35.01 |
7:  iteration     3280/   11269 | consumed samples:       839680 | consumed tokens:   1719664640 | elapsed time per iteration (s): 0.47 | learning rate: 1.665E-04 | global batch size:   256 | lm loss: 3.669516E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.864 | TFLOPs: 35.44 |
7:  iteration     3290/   11269 | consumed samples:       842240 | consumed tokens:   1724907520 | elapsed time per iteration (s): 0.47 | learning rate: 1.663E-04 | global batch size:   256 | lm loss: 3.668855E+00 | grad norm: 0.333 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.729 | TFLOPs: 35.43 |
7:  iteration     3300/   11269 | consumed samples:       844800 | consumed tokens:   1730150400 | elapsed time per iteration (s): 0.48 | learning rate: 1.661E-04 | global batch size:   256 | lm loss: 3.668782E+00 | grad norm: 0.284 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.951 | TFLOPs: 34.79 |
7:  iteration     3310/   11269 | consumed samples:       847360 | consumed tokens:   1735393280 | elapsed time per iteration (s): 0.47 | learning rate: 1.659E-04 | global batch size:   256 | lm loss: 3.662261E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.766 | TFLOPs: 35.30 |
7:  iteration     3320/   11269 | consumed samples:       849920 | consumed tokens:   1740636160 | elapsed time per iteration (s): 0.47 | learning rate: 1.657E-04 | global batch size:   256 | lm loss: 3.658845E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.253 | TFLOPs: 35.46 |
7:  iteration     3330/   11269 | consumed samples:       852480 | consumed tokens:   1745879040 | elapsed time per iteration (s): 0.47 | learning rate: 1.655E-04 | global batch size:   256 | lm loss: 3.656601E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.050 | TFLOPs: 35.45 |
7:  iteration     3340/   11269 | consumed samples:       855040 | consumed tokens:   1751121920 | elapsed time per iteration (s): 0.47 | learning rate: 1.653E-04 | global batch size:   256 | lm loss: 3.655730E+00 | grad norm: 0.350 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.196 | TFLOPs: 35.46 |
7:  iteration     3350/   11269 | consumed samples:       857600 | consumed tokens:   1756364800 | elapsed time per iteration (s): 0.47 | learning rate: 1.651E-04 | global batch size:   256 | lm loss: 3.646388E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.725 | TFLOPs: 35.30 |
7:  iteration     3360/   11269 | consumed samples:       860160 | consumed tokens:   1761607680 | elapsed time per iteration (s): 0.47 | learning rate: 1.649E-04 | global batch size:   256 | lm loss: 3.641381E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.328 | TFLOPs: 35.47 |
7:  iteration     3370/   11269 | consumed samples:       862720 | consumed tokens:   1766850560 | elapsed time per iteration (s): 0.47 | learning rate: 1.647E-04 | global batch size:   256 | lm loss: 3.639558E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.160 | TFLOPs: 35.46 |
7:  iteration     3380/   11269 | consumed samples:       865280 | consumed tokens:   1772093440 | elapsed time per iteration (s): 0.47 | learning rate: 1.645E-04 | global batch size:   256 | lm loss: 3.639384E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.007 | TFLOPs: 35.45 |
7:  iteration     3390/   11269 | consumed samples:       867840 | consumed tokens:   1777336320 | elapsed time per iteration (s): 0.47 | learning rate: 1.643E-04 | global batch size:   256 | lm loss: 3.643041E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.071 | TFLOPs: 35.45 |
7:  iteration     3400/   11269 | consumed samples:       870400 | consumed tokens:   1782579200 | elapsed time per iteration (s): 0.48 | learning rate: 1.641E-04 | global batch size:   256 | lm loss: 3.646726E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.854 | TFLOPs: 34.98 |
7:  iteration     3410/   11269 | consumed samples:       872960 | consumed tokens:   1787822080 | elapsed time per iteration (s): 0.47 | learning rate: 1.639E-04 | global batch size:   256 | lm loss: 3.629934E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.598 | TFLOPs: 35.29 |
7:  iteration     3420/   11269 | consumed samples:       875520 | consumed tokens:   1793064960 | elapsed time per iteration (s): 0.47 | learning rate: 1.637E-04 | global batch size:   256 | lm loss: 3.628138E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.195 | TFLOPs: 35.46 |
7:  iteration     3430/   11269 | consumed samples:       878080 | consumed tokens:   1798307840 | elapsed time per iteration (s): 0.47 | learning rate: 1.635E-04 | global batch size:   256 | lm loss: 3.640280E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.155 | TFLOPs: 35.46 |
7:  iteration     3440/   11269 | consumed samples:       880640 | consumed tokens:   1803550720 | elapsed time per iteration (s): 0.47 | learning rate: 1.633E-04 | global batch size:   256 | lm loss: 3.625219E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.171 | TFLOPs: 35.46 |
7:  iteration     3450/   11269 | consumed samples:       883200 | consumed tokens:   1808793600 | elapsed time per iteration (s): 0.47 | learning rate: 1.631E-04 | global batch size:   256 | lm loss: 3.636771E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.099 | TFLOPs: 35.45 |
7:  iteration     3460/   11269 | consumed samples:       885760 | consumed tokens:   1814036480 | elapsed time per iteration (s): 0.48 | learning rate: 1.629E-04 | global batch size:   256 | lm loss: 3.630505E+00 | grad norm: 0.354 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.837 | TFLOPs: 35.11 |
7:  iteration     3470/   11269 | consumed samples:       888320 | consumed tokens:   1819279360 | elapsed time per iteration (s): 0.47 | learning rate: 1.627E-04 | global batch size:   256 | lm loss: 3.635820E+00 | grad norm: 0.353 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.008 | TFLOPs: 35.45 |
7:  iteration     3480/   11269 | consumed samples:       890880 | consumed tokens:   1824522240 | elapsed time per iteration (s): 0.47 | learning rate: 1.625E-04 | global batch size:   256 | lm loss: 3.614988E+00 | grad norm: 0.340 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.597 | TFLOPs: 35.42 |
7:  iteration     3490/   11269 | consumed samples:       893440 | consumed tokens:   1829765120 | elapsed time per iteration (s): 0.47 | learning rate: 1.623E-04 | global batch size:   256 | lm loss: 3.632345E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.743 | TFLOPs: 35.43 |
7:  iteration     3500/   11269 | consumed samples:       896000 | consumed tokens:   1835008000 | elapsed time per iteration (s): 0.47 | learning rate: 1.621E-04 | global batch size:   256 | lm loss: 3.626513E+00 | grad norm: 0.351 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.653 | TFLOPs: 35.42 |
7:  iteration     3510/   11269 | consumed samples:       898560 | consumed tokens:   1840250880 | elapsed time per iteration (s): 0.47 | learning rate: 1.619E-04 | global batch size:   256 | lm loss: 3.622261E+00 | grad norm: 0.333 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.650 | TFLOPs: 35.42 |
7:  iteration     3520/   11269 | consumed samples:       901120 | consumed tokens:   1845493760 | elapsed time per iteration (s): 0.48 | learning rate: 1.617E-04 | global batch size:   256 | lm loss: 3.619210E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.384 | TFLOPs: 35.08 |
7:  iteration     3530/   11269 | consumed samples:       903680 | consumed tokens:   1850736640 | elapsed time per iteration (s): 0.48 | learning rate: 1.614E-04 | global batch size:   256 | lm loss: 3.615298E+00 | grad norm: 0.344 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.359 | TFLOPs: 35.14 |
7:  iteration     3540/   11269 | consumed samples:       906240 | consumed tokens:   1855979520 | elapsed time per iteration (s): 0.47 | learning rate: 1.612E-04 | global batch size:   256 | lm loss: 3.622126E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.157 | TFLOPs: 35.46 |
7:  iteration     3550/   11269 | consumed samples:       908800 | consumed tokens:   1861222400 | elapsed time per iteration (s): 0.47 | learning rate: 1.610E-04 | global batch size:   256 | lm loss: 3.620321E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.764 | TFLOPs: 35.43 |
7:  iteration     3560/   11269 | consumed samples:       911360 | consumed tokens:   1866465280 | elapsed time per iteration (s): 0.47 | learning rate: 1.608E-04 | global batch size:   256 | lm loss: 3.624717E+00 | grad norm: 0.376 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.903 | TFLOPs: 35.44 |
7:  iteration     3570/   11269 | consumed samples:       913920 | consumed tokens:   1871708160 | elapsed time per iteration (s): 0.47 | learning rate: 1.606E-04 | global batch size:   256 | lm loss: 3.610326E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.639 | TFLOPs: 35.42 |
7:  iteration     3580/   11269 | consumed samples:       916480 | consumed tokens:   1876951040 | elapsed time per iteration (s): 0.47 | learning rate: 1.604E-04 | global batch size:   256 | lm loss: 3.621282E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.695 | TFLOPs: 35.43 |
7:  iteration     3590/   11269 | consumed samples:       919040 | consumed tokens:   1882193920 | elapsed time per iteration (s): 0.47 | learning rate: 1.602E-04 | global batch size:   256 | lm loss: 3.622921E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.173 | TFLOPs: 35.39 |
7:  iteration     3600/   11269 | consumed samples:       921600 | consumed tokens:   1887436800 | elapsed time per iteration (s): 0.47 | learning rate: 1.600E-04 | global batch size:   256 | lm loss: 3.613615E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.905 | TFLOPs: 35.38 |
7:  iteration     3610/   11269 | consumed samples:       924160 | consumed tokens:   1892679680 | elapsed time per iteration (s): 0.47 | learning rate: 1.598E-04 | global batch size:   256 | lm loss: 3.616655E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.945 | TFLOPs: 35.31 |
7:  iteration     3620/   11269 | consumed samples:       926720 | consumed tokens:   1897922560 | elapsed time per iteration (s): 0.47 | learning rate: 1.596E-04 | global batch size:   256 | lm loss: 3.602882E+00 | grad norm: 0.333 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.331 | TFLOPs: 35.40 |
7:  iteration     3630/   11269 | consumed samples:       929280 | consumed tokens:   1903165440 | elapsed time per iteration (s): 0.47 | learning rate: 1.593E-04 | global batch size:   256 | lm loss: 3.615614E+00 | grad norm: 0.356 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.247 | TFLOPs: 35.40 |
7:  iteration     3640/   11269 | consumed samples:       931840 | consumed tokens:   1908408320 | elapsed time per iteration (s): 0.47 | learning rate: 1.591E-04 | global batch size:   256 | lm loss: 3.592681E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.769 | TFLOPs: 35.30 |
7:  iteration     3650/   11269 | consumed samples:       934400 | consumed tokens:   1913651200 | elapsed time per iteration (s): 0.48 | learning rate: 1.589E-04 | global batch size:   256 | lm loss: 3.599482E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.956 | TFLOPs: 34.79 |
7:  iteration     3660/   11269 | consumed samples:       936960 | consumed tokens:   1918894080 | elapsed time per iteration (s): 0.48 | learning rate: 1.587E-04 | global batch size:   256 | lm loss: 3.617565E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.636 | TFLOPs: 35.23 |
7:  iteration     3670/   11269 | consumed samples:       939520 | consumed tokens:   1924136960 | elapsed time per iteration (s): 0.47 | learning rate: 1.585E-04 | global batch size:   256 | lm loss: 3.602139E+00 | grad norm: 0.337 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.528 | TFLOPs: 35.42 |
7:  iteration     3680/   11269 | consumed samples:       942080 | consumed tokens:   1929379840 | elapsed time per iteration (s): 0.48 | learning rate: 1.583E-04 | global batch size:   256 | lm loss: 3.603326E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.456 | TFLOPs: 35.21 |
7:  iteration     3690/   11269 | consumed samples:       944640 | consumed tokens:   1934622720 | elapsed time per iteration (s): 0.47 | learning rate: 1.581E-04 | global batch size:   256 | lm loss: 3.620446E+00 | grad norm: 0.335 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.139 | TFLOPs: 35.39 |
7:  iteration     3700/   11269 | consumed samples:       947200 | consumed tokens:   1939865600 | elapsed time per iteration (s): 0.47 | learning rate: 1.579E-04 | global batch size:   256 | lm loss: 3.619298E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.968 | TFLOPs: 35.38 |
7:  iteration     3710/   11269 | consumed samples:       949760 | consumed tokens:   1945108480 | elapsed time per iteration (s): 0.48 | learning rate: 1.576E-04 | global batch size:   256 | lm loss: 3.591425E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 530.207 | TFLOPs: 34.68 |
7:  iteration     3720/   11269 | consumed samples:       952320 | consumed tokens:   1950351360 | elapsed time per iteration (s): 0.47 | learning rate: 1.574E-04 | global batch size:   256 | lm loss: 3.612724E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.247 | TFLOPs: 35.40 |
7:  iteration     3730/   11269 | consumed samples:       954880 | consumed tokens:   1955594240 | elapsed time per iteration (s): 0.47 | learning rate: 1.572E-04 | global batch size:   256 | lm loss: 3.596725E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.081 | TFLOPs: 35.39 |
7:  iteration     3740/   11269 | consumed samples:       957440 | consumed tokens:   1960837120 | elapsed time per iteration (s): 0.47 | learning rate: 1.570E-04 | global batch size:   256 | lm loss: 3.590895E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.968 | TFLOPs: 35.31 |
7:  iteration     3750/   11269 | consumed samples:       960000 | consumed tokens:   1966080000 | elapsed time per iteration (s): 0.47 | learning rate: 1.568E-04 | global batch size:   256 | lm loss: 3.594995E+00 | grad norm: 0.344 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.177 | TFLOPs: 35.39 |
7:  iteration     3760/   11269 | consumed samples:       962560 | consumed tokens:   1971322880 | elapsed time per iteration (s): 0.47 | learning rate: 1.566E-04 | global batch size:   256 | lm loss: 3.606850E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.761 | TFLOPs: 35.37 |
7:  iteration     3770/   11269 | consumed samples:       965120 | consumed tokens:   1976565760 | elapsed time per iteration (s): 0.47 | learning rate: 1.563E-04 | global batch size:   256 | lm loss: 3.614896E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.208 | TFLOPs: 35.39 |
7:  iteration     3780/   11269 | consumed samples:       967680 | consumed tokens:   1981808640 | elapsed time per iteration (s): 0.47 | learning rate: 1.561E-04 | global batch size:   256 | lm loss: 3.595600E+00 | grad norm: 0.332 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.509 | TFLOPs: 35.35 |
7:  iteration     3790/   11269 | consumed samples:       970240 | consumed tokens:   1987051520 | elapsed time per iteration (s): 0.48 | learning rate: 1.559E-04 | global batch size:   256 | lm loss: 3.592293E+00 | grad norm: 0.335 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.576 | TFLOPs: 34.96 |
7:  iteration     3800/   11269 | consumed samples:       972800 | consumed tokens:   1992294400 | elapsed time per iteration (s): 0.47 | learning rate: 1.557E-04 | global batch size:   256 | lm loss: 3.598350E+00 | grad norm: 0.352 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.900 | TFLOPs: 35.37 |
7:  iteration     3810/   11269 | consumed samples:       975360 | consumed tokens:   1997537280 | elapsed time per iteration (s): 0.47 | learning rate: 1.555E-04 | global batch size:   256 | lm loss: 3.580560E+00 | grad norm: 0.372 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.036 | TFLOPs: 35.38 |
7:  iteration     3820/   11269 | consumed samples:       977920 | consumed tokens:   2002780160 | elapsed time per iteration (s): 0.48 | learning rate: 1.553E-04 | global batch size:   256 | lm loss: 3.586628E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.391 | TFLOPs: 35.21 |
7:  iteration     3830/   11269 | consumed samples:       980480 | consumed tokens:   2008023040 | elapsed time per iteration (s): 0.48 | learning rate: 1.550E-04 | global batch size:   256 | lm loss: 3.577149E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.458 | TFLOPs: 35.15 |
7:  iteration     3840/   11269 | consumed samples:       983040 | consumed tokens:   2013265920 | elapsed time per iteration (s): 0.47 | learning rate: 1.548E-04 | global batch size:   256 | lm loss: 3.594820E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.645 | TFLOPs: 35.36 |
7:  iteration     3850/   11269 | consumed samples:       985600 | consumed tokens:   2018508800 | elapsed time per iteration (s): 0.48 | learning rate: 1.546E-04 | global batch size:   256 | lm loss: 3.582261E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.590 | TFLOPs: 35.22 |
7:  iteration     3860/   11269 | consumed samples:       988160 | consumed tokens:   2023751680 | elapsed time per iteration (s): 0.47 | learning rate: 1.544E-04 | global batch size:   256 | lm loss: 3.604624E+00 | grad norm: 0.349 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.594 | TFLOPs: 35.35 |
7:  iteration     3870/   11269 | consumed samples:       990720 | consumed tokens:   2028994560 | elapsed time per iteration (s): 0.47 | learning rate: 1.542E-04 | global batch size:   256 | lm loss: 3.580990E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.531 | TFLOPs: 35.35 |
7:  iteration     3880/   11269 | consumed samples:       993280 | consumed tokens:   2034237440 | elapsed time per iteration (s): 0.48 | learning rate: 1.539E-04 | global batch size:   256 | lm loss: 3.589249E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.407 | TFLOPs: 35.08 |
7:  iteration     3890/   11269 | consumed samples:       995840 | consumed tokens:   2039480320 | elapsed time per iteration (s): 0.47 | learning rate: 1.537E-04 | global batch size:   256 | lm loss: 3.584024E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.253 | TFLOPs: 35.33 |
7:  iteration     3900/   11269 | consumed samples:       998400 | consumed tokens:   2044723200 | elapsed time per iteration (s): 0.47 | learning rate: 1.535E-04 | global batch size:   256 | lm loss: 3.582876E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.266 | TFLOPs: 35.33 |
7:  iteration     3910/   11269 | consumed samples:      1000960 | consumed tokens:   2049966080 | elapsed time per iteration (s): 0.47 | learning rate: 1.533E-04 | global batch size:   256 | lm loss: 3.589555E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.529 | TFLOPs: 35.35 |
7:  iteration     3920/   11269 | consumed samples:      1003520 | consumed tokens:   2055208960 | elapsed time per iteration (s): 0.47 | learning rate: 1.530E-04 | global batch size:   256 | lm loss: 3.579715E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.388 | TFLOPs: 35.34 |
7:  iteration     3930/   11269 | consumed samples:      1006080 | consumed tokens:   2060451840 | elapsed time per iteration (s): 0.47 | learning rate: 1.528E-04 | global batch size:   256 | lm loss: 3.569087E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.454 | TFLOPs: 35.35 |
7:  iteration     3940/   11269 | consumed samples:      1008640 | consumed tokens:   2065694720 | elapsed time per iteration (s): 0.47 | learning rate: 1.526E-04 | global batch size:   256 | lm loss: 3.585363E+00 | grad norm: 0.333 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.154 | TFLOPs: 35.33 |
7:  iteration     3950/   11269 | consumed samples:      1011200 | consumed tokens:   2070937600 | elapsed time per iteration (s): 0.47 | learning rate: 1.524E-04 | global batch size:   256 | lm loss: 3.584088E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.157 | TFLOPs: 35.33 |
7:  iteration     3960/   11269 | consumed samples:      1013760 | consumed tokens:   2076180480 | elapsed time per iteration (s): 0.47 | learning rate: 1.522E-04 | global batch size:   256 | lm loss: 3.590179E+00 | grad norm: 0.340 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.056 | TFLOPs: 35.32 |
7:  iteration     3970/   11269 | consumed samples:      1016320 | consumed tokens:   2081423360 | elapsed time per iteration (s): 0.47 | learning rate: 1.519E-04 | global batch size:   256 | lm loss: 3.567812E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.231 | TFLOPs: 35.33 |
7:  iteration     3980/   11269 | consumed samples:      1018880 | consumed tokens:   2086666240 | elapsed time per iteration (s): 0.47 | learning rate: 1.517E-04 | global batch size:   256 | lm loss: 3.571168E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.915 | TFLOPs: 35.31 |
7:  iteration     3990/   11269 | consumed samples:      1021440 | consumed tokens:   2091909120 | elapsed time per iteration (s): 0.47 | learning rate: 1.515E-04 | global batch size:   256 | lm loss: 3.585706E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.073 | TFLOPs: 35.32 |
0: [2023-03-15 22:31:31,005] [INFO] [logging.py:68:log_dist] [Rank 0] step=4000, skipped=0, lr=[0.00015125198701273666, 0.00015125198701273666, 0.00015125198701273666], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration     4000/   11269 | consumed samples:      1024000 | consumed tokens:   2097152000 | elapsed time per iteration (s): 0.47 | learning rate: 1.513E-04 | global batch size:   256 | lm loss: 3.587613E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.811 | TFLOPs: 35.30 |
0: steps: 4000 loss: 3.5705 iter time (s): 0.473 samples/sec: 541.627
7: -----------------------------------------------------------------------------------------------
7: validation loss at iteration 4000 | lm loss value: 3.600785E+00 | lm loss PPL: 3.662696E+01 | 
7: -----------------------------------------------------------------------------------------------
0: saving checkpoint at iteration    4000 to checkpoints_280m5b9400m
0: [2023-03-15 22:31:31,185] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step4000 is begin to save!
0: [2023-03-15 22:31:31,189] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_01-model_00-model_states.pt...
0: [2023-03-15 22:31:31,299] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_01-model_00-model_states.pt.
0: [2023-03-15 22:31:31,299] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_03-model_00-model_states.pt...
0: [2023-03-15 22:31:31,324] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_03-model_00-model_states.pt.
0: [2023-03-15 22:31:31,325] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_04-model_00-model_states.pt...
0: [2023-03-15 22:31:31,349] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_04-model_00-model_states.pt.
0: [2023-03-15 22:31:31,349] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_05-model_00-model_states.pt...
0: [2023-03-15 22:31:31,373] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_05-model_00-model_states.pt.
0: [2023-03-15 22:31:31,373] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_06-model_00-model_states.pt...
0: [2023-03-15 22:31:31,397] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_06-model_00-model_states.pt.
0: [2023-03-15 22:31:31,397] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_07-model_00-model_states.pt...
0: [2023-03-15 22:31:31,421] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_07-model_00-model_states.pt.
0: [2023-03-15 22:31:31,422] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_08-model_00-model_states.pt...
0: [2023-03-15 22:31:31,446] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_08-model_00-model_states.pt.
0: [2023-03-15 22:31:31,446] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_09-model_00-model_states.pt...
0: [2023-03-15 22:31:31,470] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_09-model_00-model_states.pt.
0: [2023-03-15 22:31:31,470] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_10-model_00-model_states.pt...
0: [2023-03-15 22:31:31,494] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_10-model_00-model_states.pt.
0: [2023-03-15 22:31:31,494] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_11-model_00-model_states.pt...
0: [2023-03-15 22:31:31,518] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_11-model_00-model_states.pt.
0: [2023-03-15 22:31:31,518] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_12-model_00-model_states.pt...
0: [2023-03-15 22:31:31,542] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_12-model_00-model_states.pt.
0: [2023-03-15 22:31:31,543] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_13-model_00-model_states.pt...
0: [2023-03-15 22:31:31,567] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_13-model_00-model_states.pt.
0: [2023-03-15 22:31:31,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_14-model_00-model_states.pt...
0: [2023-03-15 22:31:31,591] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_14-model_00-model_states.pt.
0: [2023-03-15 22:31:31,591] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_15-model_00-model_states.pt...
0: [2023-03-15 22:31:31,615] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_15-model_00-model_states.pt.
0: [2023-03-15 22:31:31,615] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_16-model_00-model_states.pt...
0: [2023-03-15 22:31:31,639] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_16-model_00-model_states.pt.
0: [2023-03-15 22:31:31,639] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_17-model_00-model_states.pt...
0: [2023-03-15 22:31:31,663] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_17-model_00-model_states.pt.
0: [2023-03-15 22:31:31,664] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_18-model_00-model_states.pt...
0: [2023-03-15 22:31:31,688] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_18-model_00-model_states.pt.
0: [2023-03-15 22:31:31,688] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_19-model_00-model_states.pt...
0: [2023-03-15 22:31:31,712] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_19-model_00-model_states.pt.
0: [2023-03-15 22:31:31,712] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_20-model_00-model_states.pt...
0: [2023-03-15 22:31:31,736] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_20-model_00-model_states.pt.
0: [2023-03-15 22:31:31,736] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/layer_22-model_00-model_states.pt...
0: [2023-03-15 22:31:31,737] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/layer_22-model_00-model_states.pt.
0: [2023-03-15 22:31:31,738] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_280m5b9400m/global_step4000/mp_rank_00_model_states.pt
0: [2023-03-15 22:31:31,738] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/mp_rank_00_model_states.pt...
0: [2023-03-15 22:31:31,741] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/mp_rank_00_model_states.pt.
0: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:31:31,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:31:31,816] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:31:31,816] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-15 22:31:31,816] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
5: [2023-03-15 22:31:31,817] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:31:31,817] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-15 22:31:31,817] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
0: [2023-03-15 22:31:31,817] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:31:31,818] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:31:31,818] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-15 22:31:31,818] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
5: [2023-03-15 22:31:31,818] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:31:31,818] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-15 22:31:31,818] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
7: [2023-03-15 22:31:31,821] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:31:31,821] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-15 22:31:31,821] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
6: [2023-03-15 22:31:31,821] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:31:31,821] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-15 22:31:31,821] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
7: [2023-03-15 22:31:31,821] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:31:31,821] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-15 22:31:31,821] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
1: [2023-03-15 22:31:31,821] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:31:31,822] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-15 22:31:31,822] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
6: [2023-03-15 22:31:31,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:31:31,823] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-15 22:31:31,823] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
3: [2023-03-15 22:31:31,820] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:31:31,820] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:31:31,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:31:31,820] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-15 22:31:31,820] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
4: [2023-03-15 22:31:31,823] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
3: [2023-03-15 22:31:31,820] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
3: [2023-03-15 22:31:31,820] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
4: [2023-03-15 22:31:31,823] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
3: [2023-03-15 22:31:31,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:31:31,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:31:31,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:31:31,824] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:31:31,824] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-15 22:31:31,824] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
0: [2023-03-15 22:31:31,824] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:31:31,824] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-15 22:31:31,824] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
0: [2023-03-15 22:31:31,824] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:31:31,824] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-15 22:31:31,824] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
0: [2023-03-15 22:31:31,825] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:31:31,825] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-15 22:31:31,825] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
7: [2023-03-15 22:31:31,826] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:31:31,826] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-15 22:31:31,826] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
6: [2023-03-15 22:31:31,826] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:31:31,826] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:31:31,826] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:31:31,826] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-15 22:31:31,826] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
6: [2023-03-15 22:31:31,826] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
7: [2023-03-15 22:31:31,826] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
7: [2023-03-15 22:31:31,826] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
6: [2023-03-15 22:31:31,826] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
6: [2023-03-15 22:31:31,826] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:31:31,826] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-15 22:31:31,826] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
7: [2023-03-15 22:31:31,828] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:31:31,828] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-15 22:31:31,828] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
1: [2023-03-15 22:31:31,828] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:31:31,828] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-15 22:31:31,828] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
6: [2023-03-15 22:31:31,828] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:31:31,829] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-15 22:31:31,829] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
4: [2023-03-15 22:31:31,829] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:31:31,829] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:31:31,829] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
6: [2023-03-15 22:31:31,829] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
4: [2023-03-15 22:31:31,829] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
6: [2023-03-15 22:31:31,829] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
5: [2023-03-15 22:31:31,829] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:31:31,829] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-15 22:31:31,829] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:31:31,829] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
5: [2023-03-15 22:31:31,829] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
4: [2023-03-15 22:31:31,829] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:31:31,830] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
5: [2023-03-15 22:31:31,829] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
4: [2023-03-15 22:31:31,830] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
4: [2023-03-15 22:31:31,830] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:31:31,830] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-15 22:31:31,830] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
7: [2023-03-15 22:31:31,830] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:31:31,830] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-15 22:31:31,830] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
0: [2023-03-15 22:31:31,831] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:31:31,831] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-15 22:31:31,831] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
5: [2023-03-15 22:31:31,831] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:31:31,831] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:31:31,831] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-15 22:31:31,831] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-15 22:31:31,831] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
5: [2023-03-15 22:31:31,831] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
5: [2023-03-15 22:31:31,833] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:31:31,833] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-15 22:31:31,833] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
7: [2023-03-15 22:31:31,833] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:31:31,834] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-15 22:31:31,834] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
4: [2023-03-15 22:31:31,834] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:31:31,834] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-15 22:31:31,834] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
4: [2023-03-15 22:31:31,834] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:31:31,834] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-15 22:31:31,834] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
3: [2023-03-15 22:31:31,824] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-15 22:31:31,824] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-15 22:31:31,824] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-15 22:31:31,824] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
3: [2023-03-15 22:31:31,824] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
3: [2023-03-15 22:31:31,824] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
3: [2023-03-15 22:31:31,836] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:31:31,836] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:31:31,836] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:31:31,836] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-15 22:31:31,836] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-15 22:31:31,836] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
3: [2023-03-15 22:31:31,836] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
3: [2023-03-15 22:31:31,836] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-15 22:31:31,836] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
0: [2023-03-15 22:31:31,839] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:31:31,839] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-15 22:31:31,839] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
0: [2023-03-15 22:31:31,840] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:31:31,840] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-15 22:31:31,840] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
2: [2023-03-15 22:31:31,821] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:31:31,821] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-15 22:31:31,821] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
2: [2023-03-15 22:31:31,822] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:31:31,822] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-15 22:31:31,822] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
2: [2023-03-15 22:31:31,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:31:31,823] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-15 22:31:31,823] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
2: [2023-03-15 22:31:31,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:31:31,823] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-15 22:31:31,823] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
2: [2023-03-15 22:31:31,828] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:31:31,828] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-15 22:31:31,828] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
2: [2023-03-15 22:31:31,833] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:31:31,833] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-15 22:31:31,833] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
6: [2023-03-15 22:31:31,843] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:31:31,843] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-15 22:31:31,843] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
5: [2023-03-15 22:31:31,846] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:31:31,842] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:31:31,842] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:31:31,842] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-15 22:31:31,842] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-15 22:31:31,842] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
2: [2023-03-15 22:31:31,842] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
5: [2023-03-15 22:31:31,846] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-15 22:31:31,846] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
1: [2023-03-15 22:31:31,847] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:31:31,848] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-15 22:31:31,848] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
6: [2023-03-15 22:31:31,849] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:31:31,849] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-15 22:31:31,849] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
1: [2023-03-15 22:31:31,854] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:31:31,854] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-15 22:31:31,854] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
1: [2023-03-15 22:31:31,892] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:31:31,892] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-15 22:31:31,892] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
1: [2023-03-15 22:31:31,902] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:31:31,902] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-15 22:31:31,902] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
1: [2023-03-15 22:31:31,909] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:31:31,910] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-15 22:31:31,910] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
0: [2023-03-15 22:31:31,914] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-15 22:31:31,914] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
1: [2023-03-15 22:31:31,919] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:31:31,919] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step4000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-15 22:31:31,919] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step4000 is ready now!
0:   successfully saved checkpoint at iteration    4000 to checkpoints_280m5b9400m
7: time (ms) | save-checkpoint: 740.19
7:  iteration     4010/   11269 | consumed samples:      1026560 | consumed tokens:   2102394880 | elapsed time per iteration (s): 0.56 | learning rate: 1.510E-04 | global batch size:   256 | lm loss: 3.569915E+00 | grad norm: 0.357 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 455.525 | TFLOPs: 29.79 |
7:  iteration     4020/   11269 | consumed samples:      1029120 | consumed tokens:   2107637760 | elapsed time per iteration (s): 0.47 | learning rate: 1.508E-04 | global batch size:   256 | lm loss: 3.572099E+00 | grad norm: 0.374 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.834 | TFLOPs: 35.44 |
7:  iteration     4030/   11269 | consumed samples:      1031680 | consumed tokens:   2112880640 | elapsed time per iteration (s): 0.47 | learning rate: 1.506E-04 | global batch size:   256 | lm loss: 3.578250E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.100 | TFLOPs: 35.39 |
7:  iteration     4040/   11269 | consumed samples:      1034240 | consumed tokens:   2118123520 | elapsed time per iteration (s): 0.48 | learning rate: 1.503E-04 | global batch size:   256 | lm loss: 3.570599E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.790 | TFLOPs: 35.04 |
7:  iteration     4050/   11269 | consumed samples:      1036800 | consumed tokens:   2123366400 | elapsed time per iteration (s): 0.47 | learning rate: 1.501E-04 | global batch size:   256 | lm loss: 3.563543E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.827 | TFLOPs: 35.37 |
7:  iteration     4060/   11269 | consumed samples:      1039360 | consumed tokens:   2128609280 | elapsed time per iteration (s): 0.47 | learning rate: 1.499E-04 | global batch size:   256 | lm loss: 3.585852E+00 | grad norm: 0.326 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.273 | TFLOPs: 35.33 |
7:  iteration     4070/   11269 | consumed samples:      1041920 | consumed tokens:   2133852160 | elapsed time per iteration (s): 0.47 | learning rate: 1.497E-04 | global batch size:   256 | lm loss: 3.576198E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.390 | TFLOPs: 35.34 |
7:  iteration     4080/   11269 | consumed samples:      1044480 | consumed tokens:   2139095040 | elapsed time per iteration (s): 0.47 | learning rate: 1.494E-04 | global batch size:   256 | lm loss: 3.573927E+00 | grad norm: 0.329 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.248 | TFLOPs: 35.33 |
7:  iteration     4090/   11269 | consumed samples:      1047040 | consumed tokens:   2144337920 | elapsed time per iteration (s): 0.47 | learning rate: 1.492E-04 | global batch size:   256 | lm loss: 3.561845E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.957 | TFLOPs: 35.31 |
7:  iteration     4100/   11269 | consumed samples:      1049600 | consumed tokens:   2149580800 | elapsed time per iteration (s): 0.47 | learning rate: 1.490E-04 | global batch size:   256 | lm loss: 3.567532E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.822 | TFLOPs: 35.30 |
7:  iteration     4110/   11269 | consumed samples:      1052160 | consumed tokens:   2154823680 | elapsed time per iteration (s): 0.47 | learning rate: 1.488E-04 | global batch size:   256 | lm loss: 3.561104E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.639 | TFLOPs: 35.29 |
7:  iteration     4120/   11269 | consumed samples:      1054720 | consumed tokens:   2160066560 | elapsed time per iteration (s): 0.48 | learning rate: 1.485E-04 | global batch size:   256 | lm loss: 3.549779E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.892 | TFLOPs: 35.11 |
7:  iteration     4130/   11269 | consumed samples:      1057280 | consumed tokens:   2165309440 | elapsed time per iteration (s): 0.48 | learning rate: 1.483E-04 | global batch size:   256 | lm loss: 3.555961E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.164 | TFLOPs: 35.20 |
7:  iteration     4140/   11269 | consumed samples:      1059840 | consumed tokens:   2170552320 | elapsed time per iteration (s): 0.47 | learning rate: 1.481E-04 | global batch size:   256 | lm loss: 3.573557E+00 | grad norm: 0.354 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.848 | TFLOPs: 35.31 |
7:  iteration     4150/   11269 | consumed samples:      1062400 | consumed tokens:   2175795200 | elapsed time per iteration (s): 0.47 | learning rate: 1.478E-04 | global batch size:   256 | lm loss: 3.556122E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.884 | TFLOPs: 35.31 |
7:  iteration     4160/   11269 | consumed samples:      1064960 | consumed tokens:   2181038080 | elapsed time per iteration (s): 0.47 | learning rate: 1.476E-04 | global batch size:   256 | lm loss: 3.564204E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.586 | TFLOPs: 35.29 |
7:  iteration     4170/   11269 | consumed samples:      1067520 | consumed tokens:   2186280960 | elapsed time per iteration (s): 0.47 | learning rate: 1.474E-04 | global batch size:   256 | lm loss: 3.555713E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.770 | TFLOPs: 35.30 |
7:  iteration     4180/   11269 | consumed samples:      1070080 | consumed tokens:   2191523840 | elapsed time per iteration (s): 0.47 | learning rate: 1.471E-04 | global batch size:   256 | lm loss: 3.556392E+00 | grad norm: 0.349 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.909 | TFLOPs: 35.31 |
7:  iteration     4190/   11269 | consumed samples:      1072640 | consumed tokens:   2196766720 | elapsed time per iteration (s): 0.47 | learning rate: 1.469E-04 | global batch size:   256 | lm loss: 3.555935E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.645 | TFLOPs: 35.29 |
7:  iteration     4200/   11269 | consumed samples:      1075200 | consumed tokens:   2202009600 | elapsed time per iteration (s): 0.48 | learning rate: 1.467E-04 | global batch size:   256 | lm loss: 3.554442E+00 | grad norm: 0.346 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.132 | TFLOPs: 34.93 |
7:  iteration     4210/   11269 | consumed samples:      1077760 | consumed tokens:   2207252480 | elapsed time per iteration (s): 0.47 | learning rate: 1.465E-04 | global batch size:   256 | lm loss: 3.555630E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.272 | TFLOPs: 35.33 |
7:  iteration     4220/   11269 | consumed samples:      1080320 | consumed tokens:   2212495360 | elapsed time per iteration (s): 0.47 | learning rate: 1.462E-04 | global batch size:   256 | lm loss: 3.558863E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.761 | TFLOPs: 35.30 |
7:  iteration     4230/   11269 | consumed samples:      1082880 | consumed tokens:   2217738240 | elapsed time per iteration (s): 0.48 | learning rate: 1.460E-04 | global batch size:   256 | lm loss: 3.549416E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.611 | TFLOPs: 35.16 |
7:  iteration     4240/   11269 | consumed samples:      1085440 | consumed tokens:   2222981120 | elapsed time per iteration (s): 0.47 | learning rate: 1.458E-04 | global batch size:   256 | lm loss: 3.539490E+00 | grad norm: 0.345 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.903 | TFLOPs: 35.31 |
7:  iteration     4250/   11269 | consumed samples:      1088000 | consumed tokens:   2228224000 | elapsed time per iteration (s): 0.47 | learning rate: 1.455E-04 | global batch size:   256 | lm loss: 3.544685E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.646 | TFLOPs: 35.29 |
7:  iteration     4260/   11269 | consumed samples:      1090560 | consumed tokens:   2233466880 | elapsed time per iteration (s): 0.47 | learning rate: 1.453E-04 | global batch size:   256 | lm loss: 3.551604E+00 | grad norm: 0.330 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.415 | TFLOPs: 35.28 |
7:  iteration     4270/   11269 | consumed samples:      1093120 | consumed tokens:   2238709760 | elapsed time per iteration (s): 0.47 | learning rate: 1.451E-04 | global batch size:   256 | lm loss: 3.545426E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.640 | TFLOPs: 35.29 |
7:  iteration     4280/   11269 | consumed samples:      1095680 | consumed tokens:   2243952640 | elapsed time per iteration (s): 0.47 | learning rate: 1.448E-04 | global batch size:   256 | lm loss: 3.535106E+00 | grad norm: 0.331 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.743 | TFLOPs: 35.30 |
7:  iteration     4290/   11269 | consumed samples:      1098240 | consumed tokens:   2249195520 | elapsed time per iteration (s): 0.47 | learning rate: 1.446E-04 | global batch size:   256 | lm loss: 3.549684E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.670 | TFLOPs: 35.29 |
7:  iteration     4300/   11269 | consumed samples:      1100800 | consumed tokens:   2254438400 | elapsed time per iteration (s): 0.47 | learning rate: 1.444E-04 | global batch size:   256 | lm loss: 3.540063E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.047 | TFLOPs: 35.25 |
7:  iteration     4310/   11269 | consumed samples:      1103360 | consumed tokens:   2259681280 | elapsed time per iteration (s): 0.47 | learning rate: 1.441E-04 | global batch size:   256 | lm loss: 3.545801E+00 | grad norm: 0.326 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.841 | TFLOPs: 35.31 |
7:  iteration     4320/   11269 | consumed samples:      1105920 | consumed tokens:   2264924160 | elapsed time per iteration (s): 0.47 | learning rate: 1.439E-04 | global batch size:   256 | lm loss: 3.543862E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.439 | TFLOPs: 35.28 |
7:  iteration     4330/   11269 | consumed samples:      1108480 | consumed tokens:   2270167040 | elapsed time per iteration (s): 0.47 | learning rate: 1.437E-04 | global batch size:   256 | lm loss: 3.544177E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.610 | TFLOPs: 35.29 |
7:  iteration     4340/   11269 | consumed samples:      1111040 | consumed tokens:   2275409920 | elapsed time per iteration (s): 0.47 | learning rate: 1.434E-04 | global batch size:   256 | lm loss: 3.553367E+00 | grad norm: 0.331 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.605 | TFLOPs: 35.29 |
7:  iteration     4350/   11269 | consumed samples:      1113600 | consumed tokens:   2280652800 | elapsed time per iteration (s): 0.47 | learning rate: 1.432E-04 | global batch size:   256 | lm loss: 3.532352E+00 | grad norm: 0.330 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.604 | TFLOPs: 35.29 |
7:  iteration     4360/   11269 | consumed samples:      1116160 | consumed tokens:   2285895680 | elapsed time per iteration (s): 0.47 | learning rate: 1.429E-04 | global batch size:   256 | lm loss: 3.546780E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.665 | TFLOPs: 35.29 |
7:  iteration     4370/   11269 | consumed samples:      1118720 | consumed tokens:   2291138560 | elapsed time per iteration (s): 0.47 | learning rate: 1.427E-04 | global batch size:   256 | lm loss: 3.552824E+00 | grad norm: 0.339 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.458 | TFLOPs: 35.28 |
7:  iteration     4380/   11269 | consumed samples:      1121280 | consumed tokens:   2296381440 | elapsed time per iteration (s): 0.48 | learning rate: 1.425E-04 | global batch size:   256 | lm loss: 3.532155E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.664 | TFLOPs: 35.10 |
7:  iteration     4390/   11269 | consumed samples:      1123840 | consumed tokens:   2301624320 | elapsed time per iteration (s): 0.47 | learning rate: 1.422E-04 | global batch size:   256 | lm loss: 3.544703E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.462 | TFLOPs: 35.28 |
7:  iteration     4400/   11269 | consumed samples:      1126400 | consumed tokens:   2306867200 | elapsed time per iteration (s): 0.47 | learning rate: 1.420E-04 | global batch size:   256 | lm loss: 3.547324E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.447 | TFLOPs: 35.28 |
7:  iteration     4410/   11269 | consumed samples:      1128960 | consumed tokens:   2312110080 | elapsed time per iteration (s): 0.47 | learning rate: 1.418E-04 | global batch size:   256 | lm loss: 3.548166E+00 | grad norm: 0.349 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.758 | TFLOPs: 35.30 |
7:  iteration     4420/   11269 | consumed samples:      1131520 | consumed tokens:   2317352960 | elapsed time per iteration (s): 0.47 | learning rate: 1.415E-04 | global batch size:   256 | lm loss: 3.565414E+00 | grad norm: 0.338 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.364 | TFLOPs: 35.27 |
7:  iteration     4430/   11269 | consumed samples:      1134080 | consumed tokens:   2322595840 | elapsed time per iteration (s): 0.48 | learning rate: 1.413E-04 | global batch size:   256 | lm loss: 3.541821E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.219 | TFLOPs: 35.07 |
7:  iteration     4440/   11269 | consumed samples:      1136640 | consumed tokens:   2327838720 | elapsed time per iteration (s): 0.48 | learning rate: 1.411E-04 | global batch size:   256 | lm loss: 3.532368E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.468 | TFLOPs: 35.02 |
7:  iteration     4450/   11269 | consumed samples:      1139200 | consumed tokens:   2333081600 | elapsed time per iteration (s): 0.47 | learning rate: 1.408E-04 | global batch size:   256 | lm loss: 3.527050E+00 | grad norm: 0.327 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.707 | TFLOPs: 35.30 |
7:  iteration     4460/   11269 | consumed samples:      1141760 | consumed tokens:   2338324480 | elapsed time per iteration (s): 0.47 | learning rate: 1.406E-04 | global batch size:   256 | lm loss: 3.536263E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.807 | TFLOPs: 35.30 |
7:  iteration     4470/   11269 | consumed samples:      1144320 | consumed tokens:   2343567360 | elapsed time per iteration (s): 0.47 | learning rate: 1.403E-04 | global batch size:   256 | lm loss: 3.523137E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.730 | TFLOPs: 35.30 |
7:  iteration     4480/   11269 | consumed samples:      1146880 | consumed tokens:   2348810240 | elapsed time per iteration (s): 0.47 | learning rate: 1.401E-04 | global batch size:   256 | lm loss: 3.542071E+00 | grad norm: 0.341 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.785 | TFLOPs: 35.30 |
7:  iteration     4490/   11269 | consumed samples:      1149440 | consumed tokens:   2354053120 | elapsed time per iteration (s): 0.47 | learning rate: 1.399E-04 | global batch size:   256 | lm loss: 3.519309E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.673 | TFLOPs: 35.29 |
7:  iteration     4500/   11269 | consumed samples:      1152000 | consumed tokens:   2359296000 | elapsed time per iteration (s): 0.47 | learning rate: 1.396E-04 | global batch size:   256 | lm loss: 3.535428E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.822 | TFLOPs: 35.30 |
7:  iteration     4510/   11269 | consumed samples:      1154560 | consumed tokens:   2364538880 | elapsed time per iteration (s): 0.47 | learning rate: 1.394E-04 | global batch size:   256 | lm loss: 3.532586E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.692 | TFLOPs: 35.30 |
7:  iteration     4520/   11269 | consumed samples:      1157120 | consumed tokens:   2369781760 | elapsed time per iteration (s): 0.47 | learning rate: 1.391E-04 | global batch size:   256 | lm loss: 3.536242E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.320 | TFLOPs: 35.27 |
7:  iteration     4530/   11269 | consumed samples:      1159680 | consumed tokens:   2375024640 | elapsed time per iteration (s): 0.47 | learning rate: 1.389E-04 | global batch size:   256 | lm loss: 3.544091E+00 | grad norm: 0.361 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.468 | TFLOPs: 35.28 |
7:  iteration     4540/   11269 | consumed samples:      1162240 | consumed tokens:   2380267520 | elapsed time per iteration (s): 0.47 | learning rate: 1.387E-04 | global batch size:   256 | lm loss: 3.528858E+00 | grad norm: 0.332 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.291 | TFLOPs: 35.27 |
7:  iteration     4550/   11269 | consumed samples:      1164800 | consumed tokens:   2385510400 | elapsed time per iteration (s): 0.48 | learning rate: 1.384E-04 | global batch size:   256 | lm loss: 3.529008E+00 | grad norm: 0.343 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.068 | TFLOPs: 34.99 |
7:  iteration     4560/   11269 | consumed samples:      1167360 | consumed tokens:   2390753280 | elapsed time per iteration (s): 0.47 | learning rate: 1.382E-04 | global batch size:   256 | lm loss: 3.531031E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.344 | TFLOPs: 35.27 |
7:  iteration     4570/   11269 | consumed samples:      1169920 | consumed tokens:   2395996160 | elapsed time per iteration (s): 0.47 | learning rate: 1.379E-04 | global batch size:   256 | lm loss: 3.543891E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.420 | TFLOPs: 35.28 |
7:  iteration     4580/   11269 | consumed samples:      1172480 | consumed tokens:   2401239040 | elapsed time per iteration (s): 0.47 | learning rate: 1.377E-04 | global batch size:   256 | lm loss: 3.542534E+00 | grad norm: 0.331 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.240 | TFLOPs: 35.27 |
7:  iteration     4590/   11269 | consumed samples:      1175040 | consumed tokens:   2406481920 | elapsed time per iteration (s): 0.47 | learning rate: 1.375E-04 | global batch size:   256 | lm loss: 3.544521E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.590 | TFLOPs: 35.29 |
7:  iteration     4600/   11269 | consumed samples:      1177600 | consumed tokens:   2411724800 | elapsed time per iteration (s): 0.47 | learning rate: 1.372E-04 | global batch size:   256 | lm loss: 3.532931E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.364 | TFLOPs: 35.27 |
7:  iteration     4610/   11269 | consumed samples:      1180160 | consumed tokens:   2416967680 | elapsed time per iteration (s): 0.47 | learning rate: 1.370E-04 | global batch size:   256 | lm loss: 3.525865E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.679 | TFLOPs: 35.29 |
7:  iteration     4620/   11269 | consumed samples:      1182720 | consumed tokens:   2422210560 | elapsed time per iteration (s): 0.47 | learning rate: 1.367E-04 | global batch size:   256 | lm loss: 3.537323E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.268 | TFLOPs: 35.27 |
7:  iteration     4630/   11269 | consumed samples:      1185280 | consumed tokens:   2427453440 | elapsed time per iteration (s): 0.48 | learning rate: 1.365E-04 | global batch size:   256 | lm loss: 3.534953E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.017 | TFLOPs: 34.99 |
7:  iteration     4640/   11269 | consumed samples:      1187840 | consumed tokens:   2432696320 | elapsed time per iteration (s): 0.48 | learning rate: 1.362E-04 | global batch size:   256 | lm loss: 3.510226E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.650 | TFLOPs: 35.16 |
7:  iteration     4650/   11269 | consumed samples:      1190400 | consumed tokens:   2437939200 | elapsed time per iteration (s): 0.48 | learning rate: 1.360E-04 | global batch size:   256 | lm loss: 3.534689E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.242 | TFLOPs: 34.87 |
7:  iteration     4660/   11269 | consumed samples:      1192960 | consumed tokens:   2443182080 | elapsed time per iteration (s): 0.49 | learning rate: 1.358E-04 | global batch size:   256 | lm loss: 3.531736E+00 | grad norm: 0.337 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 521.985 | TFLOPs: 34.14 |
7:  iteration     4670/   11269 | consumed samples:      1195520 | consumed tokens:   2448424960 | elapsed time per iteration (s): 0.47 | learning rate: 1.355E-04 | global batch size:   256 | lm loss: 3.505529E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.773 | TFLOPs: 35.30 |
7:  iteration     4680/   11269 | consumed samples:      1198080 | consumed tokens:   2453667840 | elapsed time per iteration (s): 0.47 | learning rate: 1.353E-04 | global batch size:   256 | lm loss: 3.498239E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.323 | TFLOPs: 35.27 |
7:  iteration     4690/   11269 | consumed samples:      1200640 | consumed tokens:   2458910720 | elapsed time per iteration (s): 0.48 | learning rate: 1.350E-04 | global batch size:   256 | lm loss: 3.515582E+00 | grad norm: 0.332 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.806 | TFLOPs: 34.98 |
7:  iteration     4700/   11269 | consumed samples:      1203200 | consumed tokens:   2464153600 | elapsed time per iteration (s): 0.47 | learning rate: 1.348E-04 | global batch size:   256 | lm loss: 3.510083E+00 | grad norm: 0.330 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.585 | TFLOPs: 35.29 |
7:  iteration     4710/   11269 | consumed samples:      1205760 | consumed tokens:   2469396480 | elapsed time per iteration (s): 0.47 | learning rate: 1.345E-04 | global batch size:   256 | lm loss: 3.524300E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.592 | TFLOPs: 35.29 |
7:  iteration     4720/   11269 | consumed samples:      1208320 | consumed tokens:   2474639360 | elapsed time per iteration (s): 0.47 | learning rate: 1.343E-04 | global batch size:   256 | lm loss: 3.517653E+00 | grad norm: 0.327 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.095 | TFLOPs: 35.26 |
7:  iteration     4730/   11269 | consumed samples:      1210880 | consumed tokens:   2479882240 | elapsed time per iteration (s): 0.48 | learning rate: 1.341E-04 | global batch size:   256 | lm loss: 3.516909E+00 | grad norm: 0.320 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.892 | TFLOPs: 35.24 |
7:  iteration     4740/   11269 | consumed samples:      1213440 | consumed tokens:   2485125120 | elapsed time per iteration (s): 0.48 | learning rate: 1.338E-04 | global batch size:   256 | lm loss: 3.514516E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.894 | TFLOPs: 35.24 |
7:  iteration     4750/   11269 | consumed samples:      1216000 | consumed tokens:   2490368000 | elapsed time per iteration (s): 0.47 | learning rate: 1.336E-04 | global batch size:   256 | lm loss: 3.507392E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.210 | TFLOPs: 35.26 |
7:  iteration     4760/   11269 | consumed samples:      1218560 | consumed tokens:   2495610880 | elapsed time per iteration (s): 0.48 | learning rate: 1.333E-04 | global batch size:   256 | lm loss: 3.513135E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.702 | TFLOPs: 35.23 |
7:  iteration     4770/   11269 | consumed samples:      1221120 | consumed tokens:   2500853760 | elapsed time per iteration (s): 0.48 | learning rate: 1.331E-04 | global batch size:   256 | lm loss: 3.512903E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.730 | TFLOPs: 35.23 |
7:  iteration     4780/   11269 | consumed samples:      1223680 | consumed tokens:   2506096640 | elapsed time per iteration (s): 0.48 | learning rate: 1.328E-04 | global batch size:   256 | lm loss: 3.520626E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.797 | TFLOPs: 35.24 |
7:  iteration     4790/   11269 | consumed samples:      1226240 | consumed tokens:   2511339520 | elapsed time per iteration (s): 0.48 | learning rate: 1.326E-04 | global batch size:   256 | lm loss: 3.503631E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.875 | TFLOPs: 35.24 |
7:  iteration     4800/   11269 | consumed samples:      1228800 | consumed tokens:   2516582400 | elapsed time per iteration (s): 0.48 | learning rate: 1.323E-04 | global batch size:   256 | lm loss: 3.505391E+00 | grad norm: 0.350 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.823 | TFLOPs: 35.24 |
7:  iteration     4810/   11269 | consumed samples:      1231360 | consumed tokens:   2521825280 | elapsed time per iteration (s): 0.48 | learning rate: 1.321E-04 | global batch size:   256 | lm loss: 3.507911E+00 | grad norm: 0.363 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.733 | TFLOPs: 35.23 |
7:  iteration     4820/   11269 | consumed samples:      1233920 | consumed tokens:   2527068160 | elapsed time per iteration (s): 0.48 | learning rate: 1.319E-04 | global batch size:   256 | lm loss: 3.507335E+00 | grad norm: 0.329 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.703 | TFLOPs: 35.23 |
7:  iteration     4830/   11269 | consumed samples:      1236480 | consumed tokens:   2532311040 | elapsed time per iteration (s): 0.47 | learning rate: 1.316E-04 | global batch size:   256 | lm loss: 3.505484E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.055 | TFLOPs: 35.25 |
7:  iteration     4840/   11269 | consumed samples:      1239040 | consumed tokens:   2537553920 | elapsed time per iteration (s): 0.48 | learning rate: 1.314E-04 | global batch size:   256 | lm loss: 3.497747E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.934 | TFLOPs: 35.25 |
7:  iteration     4850/   11269 | consumed samples:      1241600 | consumed tokens:   2542796800 | elapsed time per iteration (s): 0.48 | learning rate: 1.311E-04 | global batch size:   256 | lm loss: 3.506871E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.885 | TFLOPs: 35.05 |
7:  iteration     4860/   11269 | consumed samples:      1244160 | consumed tokens:   2548039680 | elapsed time per iteration (s): 0.48 | learning rate: 1.309E-04 | global batch size:   256 | lm loss: 3.502287E+00 | grad norm: 0.335 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.918 | TFLOPs: 35.25 |
7:  iteration     4870/   11269 | consumed samples:      1246720 | consumed tokens:   2553282560 | elapsed time per iteration (s): 0.48 | learning rate: 1.306E-04 | global batch size:   256 | lm loss: 3.509565E+00 | grad norm: 0.360 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.348 | TFLOPs: 35.21 |
7:  iteration     4880/   11269 | consumed samples:      1249280 | consumed tokens:   2558525440 | elapsed time per iteration (s): 0.48 | learning rate: 1.304E-04 | global batch size:   256 | lm loss: 3.516523E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.501 | TFLOPs: 35.22 |
7:  iteration     4890/   11269 | consumed samples:      1251840 | consumed tokens:   2563768320 | elapsed time per iteration (s): 0.48 | learning rate: 1.301E-04 | global batch size:   256 | lm loss: 3.501132E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.434 | TFLOPs: 35.21 |
7:  iteration     4900/   11269 | consumed samples:      1254400 | consumed tokens:   2569011200 | elapsed time per iteration (s): 0.48 | learning rate: 1.299E-04 | global batch size:   256 | lm loss: 3.505419E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.330 | TFLOPs: 35.21 |
7:  iteration     4910/   11269 | consumed samples:      1256960 | consumed tokens:   2574254080 | elapsed time per iteration (s): 0.48 | learning rate: 1.296E-04 | global batch size:   256 | lm loss: 3.496089E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.081 | TFLOPs: 35.19 |
7:  iteration     4920/   11269 | consumed samples:      1259520 | consumed tokens:   2579496960 | elapsed time per iteration (s): 0.48 | learning rate: 1.294E-04 | global batch size:   256 | lm loss: 3.493664E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.981 | TFLOPs: 35.18 |
7:  iteration     4930/   11269 | consumed samples:      1262080 | consumed tokens:   2584739840 | elapsed time per iteration (s): 0.48 | learning rate: 1.291E-04 | global batch size:   256 | lm loss: 3.491578E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.861 | TFLOPs: 35.18 |
7:  iteration     4940/   11269 | consumed samples:      1264640 | consumed tokens:   2589982720 | elapsed time per iteration (s): 0.48 | learning rate: 1.289E-04 | global batch size:   256 | lm loss: 3.491191E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.959 | TFLOPs: 35.18 |
7:  iteration     4950/   11269 | consumed samples:      1267200 | consumed tokens:   2595225600 | elapsed time per iteration (s): 0.48 | learning rate: 1.286E-04 | global batch size:   256 | lm loss: 3.509406E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.947 | TFLOPs: 35.18 |
7:  iteration     4960/   11269 | consumed samples:      1269760 | consumed tokens:   2600468480 | elapsed time per iteration (s): 0.48 | learning rate: 1.284E-04 | global batch size:   256 | lm loss: 3.493105E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.878 | TFLOPs: 35.18 |
7:  iteration     4970/   11269 | consumed samples:      1272320 | consumed tokens:   2605711360 | elapsed time per iteration (s): 0.48 | learning rate: 1.281E-04 | global batch size:   256 | lm loss: 3.491270E+00 | grad norm: 0.320 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.023 | TFLOPs: 35.19 |
7:  iteration     4980/   11269 | consumed samples:      1274880 | consumed tokens:   2610954240 | elapsed time per iteration (s): 0.48 | learning rate: 1.279E-04 | global batch size:   256 | lm loss: 3.498028E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.034 | TFLOPs: 35.19 |
7:  iteration     4990/   11269 | consumed samples:      1277440 | consumed tokens:   2616197120 | elapsed time per iteration (s): 0.48 | learning rate: 1.276E-04 | global batch size:   256 | lm loss: 3.509027E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.110 | TFLOPs: 35.06 |
7:  iteration     5000/   11269 | consumed samples:      1280000 | consumed tokens:   2621440000 | elapsed time per iteration (s): 0.48 | learning rate: 1.274E-04 | global batch size:   256 | lm loss: 3.503295E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.285 | TFLOPs: 35.20 |
7: -----------------------------------------------------------------------------------------------
7: validation loss at iteration 5000 | lm loss value: 3.535530E+00 | lm loss PPL: 3.431318E+01 | 
7: -----------------------------------------------------------------------------------------------
0: saving checkpoint at iteration    5000 to checkpoints_280m5b9400m
0: [2023-03-15 22:39:27,305] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step5000 is begin to save!
0: [2023-03-15 22:39:27,309] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_01-model_00-model_states.pt...
0: [2023-03-15 22:39:27,421] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_01-model_00-model_states.pt.
0: [2023-03-15 22:39:27,422] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_03-model_00-model_states.pt...
0: [2023-03-15 22:39:27,448] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_03-model_00-model_states.pt.
0: [2023-03-15 22:39:27,448] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_04-model_00-model_states.pt...
0: [2023-03-15 22:39:27,473] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_04-model_00-model_states.pt.
0: [2023-03-15 22:39:27,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_05-model_00-model_states.pt...
0: [2023-03-15 22:39:27,497] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_05-model_00-model_states.pt.
0: [2023-03-15 22:39:27,497] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_06-model_00-model_states.pt...
0: [2023-03-15 22:39:27,521] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_06-model_00-model_states.pt.
0: [2023-03-15 22:39:27,521] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_07-model_00-model_states.pt...
0: [2023-03-15 22:39:27,545] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_07-model_00-model_states.pt.
0: [2023-03-15 22:39:27,546] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_08-model_00-model_states.pt...
0: [2023-03-15 22:39:27,569] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_08-model_00-model_states.pt.
0: [2023-03-15 22:39:27,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_09-model_00-model_states.pt...
0: [2023-03-15 22:39:27,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_09-model_00-model_states.pt.
0: [2023-03-15 22:39:27,594] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_10-model_00-model_states.pt...
0: [2023-03-15 22:39:27,618] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_10-model_00-model_states.pt.
0: [2023-03-15 22:39:27,618] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_11-model_00-model_states.pt...
0: [2023-03-15 22:39:27,642] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_11-model_00-model_states.pt.
0: [2023-03-15 22:39:27,642] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_12-model_00-model_states.pt...
0: [2023-03-15 22:39:27,666] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_12-model_00-model_states.pt.
0: [2023-03-15 22:39:27,666] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_13-model_00-model_states.pt...
0: [2023-03-15 22:39:27,690] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_13-model_00-model_states.pt.
0: [2023-03-15 22:39:27,691] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_14-model_00-model_states.pt...
0: [2023-03-15 22:39:27,714] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_14-model_00-model_states.pt.
0: [2023-03-15 22:39:27,715] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_15-model_00-model_states.pt...
0: [2023-03-15 22:39:27,739] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_15-model_00-model_states.pt.
0: [2023-03-15 22:39:27,739] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_16-model_00-model_states.pt...
0: [2023-03-15 22:39:27,763] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_16-model_00-model_states.pt.
0: [2023-03-15 22:39:27,763] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_17-model_00-model_states.pt...
0: [2023-03-15 22:39:27,787] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_17-model_00-model_states.pt.
0: [2023-03-15 22:39:27,787] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_18-model_00-model_states.pt...
0: [2023-03-15 22:39:27,811] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_18-model_00-model_states.pt.
0: [2023-03-15 22:39:27,811] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_19-model_00-model_states.pt...
0: [2023-03-15 22:39:27,835] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_19-model_00-model_states.pt.
0: [2023-03-15 22:39:27,835] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_20-model_00-model_states.pt...
0: [2023-03-15 22:39:27,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_20-model_00-model_states.pt.
0: [2023-03-15 22:39:27,859] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/layer_22-model_00-model_states.pt...
0: [2023-03-15 22:39:27,860] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/layer_22-model_00-model_states.pt.
0: [2023-03-15 22:39:27,861] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_280m5b9400m/global_step5000/mp_rank_00_model_states.pt
0: [2023-03-15 22:39:27,861] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/mp_rank_00_model_states.pt...
0: [2023-03-15 22:39:27,863] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/mp_rank_00_model_states.pt.
0: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:39:27,881] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:39:27,939] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:39:27,939] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:39:27,939] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-15 22:39:27,940] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
3: [2023-03-15 22:39:27,942] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:39:27,942] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-15 22:39:27,942] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
3: [2023-03-15 22:39:27,943] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:39:27,944] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
5: [2023-03-15 22:39:27,943] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:39:27,943] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:39:27,944] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
0: [2023-03-15 22:39:27,945] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:39:27,945] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-15 22:39:27,945] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
0: [2023-03-15 22:39:27,945] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:39:27,945] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-15 22:39:27,946] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
0: [2023-03-15 22:39:27,946] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:39:27,946] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-15 22:39:27,946] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
3: [2023-03-15 22:39:27,946] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:39:27,946] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-15 22:39:27,946] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
1: [2023-03-15 22:39:27,946] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:39:27,946] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:39:27,946] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
1: [2023-03-15 22:39:27,946] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
0: [2023-03-15 22:39:27,946] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
1: [2023-03-15 22:39:27,946] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
4: [2023-03-15 22:39:27,946] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:39:27,946] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:39:27,947] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
3: [2023-03-15 22:39:27,947] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
4: [2023-03-15 22:39:27,947] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
3: [2023-03-15 22:39:27,947] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
4: [2023-03-15 22:39:27,947] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:39:27,947] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-15 22:39:27,947] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
7: [2023-03-15 22:39:27,948] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:39:27,948] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:39:27,948] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-15 22:39:27,948] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-15 22:39:27,948] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:39:27,948] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
7: [2023-03-15 22:39:27,948] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
7: [2023-03-15 22:39:27,948] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-15 22:39:27,948] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
6: [2023-03-15 22:39:27,949] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:39:27,949] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-15 22:39:27,949] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
6: [2023-03-15 22:39:27,949] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:39:27,949] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-15 22:39:27,949] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
6: [2023-03-15 22:39:27,950] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:39:27,950] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-15 22:39:27,950] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
4: [2023-03-15 22:39:27,950] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:39:27,950] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-15 22:39:27,950] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
6: [2023-03-15 22:39:27,951] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:39:27,951] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:39:27,951] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
4: [2023-03-15 22:39:27,951] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
6: [2023-03-15 22:39:27,951] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
4: [2023-03-15 22:39:27,951] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
5: [2023-03-15 22:39:27,944] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-15 22:39:27,944] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-15 22:39:27,944] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
5: [2023-03-15 22:39:27,944] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
5: [2023-03-15 22:39:27,948] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:39:27,948] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-15 22:39:27,948] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
5: [2023-03-15 22:39:27,948] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:39:27,948] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:39:27,948] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:39:27,948] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-15 22:39:27,948] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-15 22:39:27,948] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
6: [2023-03-15 22:39:27,952] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:39:27,948] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
5: [2023-03-15 22:39:27,948] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
5: [2023-03-15 22:39:27,948] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
5: [2023-03-15 22:39:27,951] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:39:27,951] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
6: [2023-03-15 22:39:27,953] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
5: [2023-03-15 22:39:27,951] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
6: [2023-03-15 22:39:27,953] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
5: [2023-03-15 22:39:27,953] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:39:27,953] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-15 22:39:27,953] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
6: [2023-03-15 22:39:27,953] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:39:27,953] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-15 22:39:27,953] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
6: [2023-03-15 22:39:27,953] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:39:27,954] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-15 22:39:27,954] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
4: [2023-03-15 22:39:27,954] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:39:27,954] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-15 22:39:27,954] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
7: [2023-03-15 22:39:27,955] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:39:27,955] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-15 22:39:27,955] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
7: [2023-03-15 22:39:27,958] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:39:27,958] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:39:27,958] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:39:27,958] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-15 22:39:27,958] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-15 22:39:27,958] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
7: [2023-03-15 22:39:27,958] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-15 22:39:27,958] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
7: [2023-03-15 22:39:27,958] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
3: [2023-03-15 22:39:27,958] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:39:27,959] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-15 22:39:27,959] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
7: [2023-03-15 22:39:27,962] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:39:27,963] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-15 22:39:27,963] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
4: [2023-03-15 22:39:27,964] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:39:27,964] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-15 22:39:27,964] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
4: [2023-03-15 22:39:27,964] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:39:27,964] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-15 22:39:27,964] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:39:27,964] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
4: [2023-03-15 22:39:27,964] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-15 22:39:27,964] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
6: [2023-03-15 22:39:27,965] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:39:27,965] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-15 22:39:27,965] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
0: [2023-03-15 22:39:27,967] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-15 22:39:27,967] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
1: [2023-03-15 22:39:27,979] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:39:27,980] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-15 22:39:27,980] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
1: [2023-03-15 22:39:27,988] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:39:27,988] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-15 22:39:27,988] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
2: [2023-03-15 22:39:27,940] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:39:27,940] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-15 22:39:27,941] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
2: [2023-03-15 22:39:27,943] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:39:27,943] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-15 22:39:27,944] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
2: [2023-03-15 22:39:27,950] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:39:27,950] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-15 22:39:27,950] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
2: [2023-03-15 22:39:27,950] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:39:27,950] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-15 22:39:27,950] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
2: [2023-03-15 22:39:27,952] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:39:27,952] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-15 22:39:27,952] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
2: [2023-03-15 22:39:27,964] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:39:27,964] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-15 22:39:27,964] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
2: [2023-03-15 22:39:27,965] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:39:27,965] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-15 22:39:27,965] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
2: [2023-03-15 22:39:27,965] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:39:27,965] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-15 22:39:27,965] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
1: [2023-03-15 22:39:27,992] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:39:27,992] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-15 22:39:27,992] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
3: [2023-03-15 22:39:27,996] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:39:27,996] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-15 22:39:27,996] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
1: [2023-03-15 22:39:28,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:39:28,003] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-15 22:39:28,003] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
1: [2023-03-15 22:39:28,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:39:28,003] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-15 22:39:28,003] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
1: [2023-03-15 22:39:28,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:39:28,004] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-15 22:39:28,004] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
3: [2023-03-15 22:39:28,022] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:39:28,022] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-15 22:39:28,022] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
1: [2023-03-15 22:39:28,024] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:39:28,024] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-15 22:39:28,024] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
0: [2023-03-15 22:39:28,030] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:39:28,030] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:39:28,030] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:39:28,030] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-15 22:39:28,030] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-15 22:39:28,030] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step5000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-15 22:39:28,030] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
0: [2023-03-15 22:39:28,030] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
0: [2023-03-15 22:39:28,030] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step5000 is ready now!
0:   successfully saved checkpoint at iteration    5000 to checkpoints_280m5b9400m
7: time (ms) | save-checkpoint: 733.99
7:  iteration     5010/   11269 | consumed samples:      1282560 | consumed tokens:   2626682880 | elapsed time per iteration (s): 0.56 | learning rate: 1.272E-04 | global batch size:   256 | lm loss: 3.493475E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 453.314 | TFLOPs: 29.65 |
7:  iteration     5020/   11269 | consumed samples:      1285120 | consumed tokens:   2631925760 | elapsed time per iteration (s): 0.47 | learning rate: 1.269E-04 | global batch size:   256 | lm loss: 3.495748E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.727 | TFLOPs: 35.30 |
7:  iteration     5030/   11269 | consumed samples:      1287680 | consumed tokens:   2637168640 | elapsed time per iteration (s): 0.47 | learning rate: 1.267E-04 | global batch size:   256 | lm loss: 3.492948E+00 | grad norm: 0.346 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.205 | TFLOPs: 35.26 |
7:  iteration     5040/   11269 | consumed samples:      1290240 | consumed tokens:   2642411520 | elapsed time per iteration (s): 0.48 | learning rate: 1.264E-04 | global batch size:   256 | lm loss: 3.490750E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.627 | TFLOPs: 35.23 |
7:  iteration     5050/   11269 | consumed samples:      1292800 | consumed tokens:   2647654400 | elapsed time per iteration (s): 0.48 | learning rate: 1.262E-04 | global batch size:   256 | lm loss: 3.484851E+00 | grad norm: 0.374 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.974 | TFLOPs: 35.18 |
7:  iteration     5060/   11269 | consumed samples:      1295360 | consumed tokens:   2652897280 | elapsed time per iteration (s): 0.48 | learning rate: 1.259E-04 | global batch size:   256 | lm loss: 3.489478E+00 | grad norm: 0.337 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.276 | TFLOPs: 35.20 |
7:  iteration     5070/   11269 | consumed samples:      1297920 | consumed tokens:   2658140160 | elapsed time per iteration (s): 0.48 | learning rate: 1.257E-04 | global batch size:   256 | lm loss: 3.500307E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.013 | TFLOPs: 35.19 |
7:  iteration     5080/   11269 | consumed samples:      1300480 | consumed tokens:   2663383040 | elapsed time per iteration (s): 0.48 | learning rate: 1.254E-04 | global batch size:   256 | lm loss: 3.504337E+00 | grad norm: 0.333 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.714 | TFLOPs: 35.17 |
7:  iteration     5090/   11269 | consumed samples:      1303040 | consumed tokens:   2668625920 | elapsed time per iteration (s): 0.48 | learning rate: 1.252E-04 | global batch size:   256 | lm loss: 3.505586E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.914 | TFLOPs: 35.18 |
7:  iteration     5100/   11269 | consumed samples:      1305600 | consumed tokens:   2673868800 | elapsed time per iteration (s): 0.48 | learning rate: 1.249E-04 | global batch size:   256 | lm loss: 3.500150E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.899 | TFLOPs: 35.18 |
7:  iteration     5110/   11269 | consumed samples:      1308160 | consumed tokens:   2679111680 | elapsed time per iteration (s): 0.48 | learning rate: 1.247E-04 | global batch size:   256 | lm loss: 3.475576E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.122 | TFLOPs: 35.19 |
7:  iteration     5120/   11269 | consumed samples:      1310720 | consumed tokens:   2684354560 | elapsed time per iteration (s): 0.48 | learning rate: 1.244E-04 | global batch size:   256 | lm loss: 3.491161E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.145 | TFLOPs: 35.19 |
7:  iteration     5130/   11269 | consumed samples:      1313280 | consumed tokens:   2689597440 | elapsed time per iteration (s): 0.48 | learning rate: 1.242E-04 | global batch size:   256 | lm loss: 3.484547E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.631 | TFLOPs: 35.16 |
7:  iteration     5140/   11269 | consumed samples:      1315840 | consumed tokens:   2694840320 | elapsed time per iteration (s): 0.48 | learning rate: 1.239E-04 | global batch size:   256 | lm loss: 3.499185E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.968 | TFLOPs: 35.18 |
7:  iteration     5150/   11269 | consumed samples:      1318400 | consumed tokens:   2700083200 | elapsed time per iteration (s): 0.48 | learning rate: 1.237E-04 | global batch size:   256 | lm loss: 3.491121E+00 | grad norm: 0.329 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.950 | TFLOPs: 35.18 |
7:  iteration     5160/   11269 | consumed samples:      1320960 | consumed tokens:   2705326080 | elapsed time per iteration (s): 0.48 | learning rate: 1.234E-04 | global batch size:   256 | lm loss: 3.479597E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.001 | TFLOPs: 35.19 |
7:  iteration     5170/   11269 | consumed samples:      1323520 | consumed tokens:   2710568960 | elapsed time per iteration (s): 0.48 | learning rate: 1.232E-04 | global batch size:   256 | lm loss: 3.468608E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.012 | TFLOPs: 35.19 |
7:  iteration     5180/   11269 | consumed samples:      1326080 | consumed tokens:   2715811840 | elapsed time per iteration (s): 0.48 | learning rate: 1.229E-04 | global batch size:   256 | lm loss: 3.480220E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.625 | TFLOPs: 35.16 |
7:  iteration     5190/   11269 | consumed samples:      1328640 | consumed tokens:   2721054720 | elapsed time per iteration (s): 0.48 | learning rate: 1.227E-04 | global batch size:   256 | lm loss: 3.484446E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.368 | TFLOPs: 34.95 |
7:  iteration     5200/   11269 | consumed samples:      1331200 | consumed tokens:   2726297600 | elapsed time per iteration (s): 0.48 | learning rate: 1.224E-04 | global batch size:   256 | lm loss: 3.499223E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.153 | TFLOPs: 35.20 |
7:  iteration     5210/   11269 | consumed samples:      1333760 | consumed tokens:   2731540480 | elapsed time per iteration (s): 0.48 | learning rate: 1.221E-04 | global batch size:   256 | lm loss: 3.490013E+00 | grad norm: 0.330 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.960 | TFLOPs: 35.18 |
7:  iteration     5220/   11269 | consumed samples:      1336320 | consumed tokens:   2736783360 | elapsed time per iteration (s): 0.48 | learning rate: 1.219E-04 | global batch size:   256 | lm loss: 3.470759E+00 | grad norm: 0.367 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.037 | TFLOPs: 34.93 |
7:  iteration     5230/   11269 | consumed samples:      1338880 | consumed tokens:   2742026240 | elapsed time per iteration (s): 0.48 | learning rate: 1.216E-04 | global batch size:   256 | lm loss: 3.484445E+00 | grad norm: 0.320 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.226 | TFLOPs: 35.20 |
7:  iteration     5240/   11269 | consumed samples:      1341440 | consumed tokens:   2747269120 | elapsed time per iteration (s): 0.48 | learning rate: 1.214E-04 | global batch size:   256 | lm loss: 3.484791E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.982 | TFLOPs: 35.18 |
7:  iteration     5250/   11269 | consumed samples:      1344000 | consumed tokens:   2752512000 | elapsed time per iteration (s): 0.48 | learning rate: 1.211E-04 | global batch size:   256 | lm loss: 3.488728E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.982 | TFLOPs: 35.18 |
7:  iteration     5260/   11269 | consumed samples:      1346560 | consumed tokens:   2757754880 | elapsed time per iteration (s): 0.48 | learning rate: 1.209E-04 | global batch size:   256 | lm loss: 3.476784E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.101 | TFLOPs: 35.19 |
7:  iteration     5270/   11269 | consumed samples:      1349120 | consumed tokens:   2762997760 | elapsed time per iteration (s): 0.48 | learning rate: 1.206E-04 | global batch size:   256 | lm loss: 3.474360E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.195 | TFLOPs: 35.20 |
7:  iteration     5280/   11269 | consumed samples:      1351680 | consumed tokens:   2768240640 | elapsed time per iteration (s): 0.48 | learning rate: 1.204E-04 | global batch size:   256 | lm loss: 3.473171E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.940 | TFLOPs: 35.18 |
7:  iteration     5290/   11269 | consumed samples:      1354240 | consumed tokens:   2773483520 | elapsed time per iteration (s): 0.48 | learning rate: 1.201E-04 | global batch size:   256 | lm loss: 3.466803E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.933 | TFLOPs: 35.18 |
7:  iteration     5300/   11269 | consumed samples:      1356800 | consumed tokens:   2778726400 | elapsed time per iteration (s): 0.48 | learning rate: 1.199E-04 | global batch size:   256 | lm loss: 3.475943E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.818 | TFLOPs: 35.17 |
7:  iteration     5310/   11269 | consumed samples:      1359360 | consumed tokens:   2783969280 | elapsed time per iteration (s): 0.48 | learning rate: 1.196E-04 | global batch size:   256 | lm loss: 3.485269E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.978 | TFLOPs: 35.18 |
7:  iteration     5320/   11269 | consumed samples:      1361920 | consumed tokens:   2789212160 | elapsed time per iteration (s): 0.48 | learning rate: 1.194E-04 | global batch size:   256 | lm loss: 3.478102E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.995 | TFLOPs: 35.18 |
7:  iteration     5330/   11269 | consumed samples:      1364480 | consumed tokens:   2794455040 | elapsed time per iteration (s): 0.48 | learning rate: 1.191E-04 | global batch size:   256 | lm loss: 3.488144E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.280 | TFLOPs: 35.20 |
7:  iteration     5340/   11269 | consumed samples:      1367040 | consumed tokens:   2799697920 | elapsed time per iteration (s): 0.48 | learning rate: 1.189E-04 | global batch size:   256 | lm loss: 3.476329E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.682 | TFLOPs: 35.16 |
7:  iteration     5350/   11269 | consumed samples:      1369600 | consumed tokens:   2804940800 | elapsed time per iteration (s): 0.48 | learning rate: 1.186E-04 | global batch size:   256 | lm loss: 3.476046E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.907 | TFLOPs: 35.18 |
7:  iteration     5360/   11269 | consumed samples:      1372160 | consumed tokens:   2810183680 | elapsed time per iteration (s): 0.48 | learning rate: 1.184E-04 | global batch size:   256 | lm loss: 3.475746E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.035 | TFLOPs: 35.19 |
7:  iteration     5370/   11269 | consumed samples:      1374720 | consumed tokens:   2815426560 | elapsed time per iteration (s): 0.48 | learning rate: 1.181E-04 | global batch size:   256 | lm loss: 3.482919E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.001 | TFLOPs: 35.19 |
7:  iteration     5380/   11269 | consumed samples:      1377280 | consumed tokens:   2820669440 | elapsed time per iteration (s): 0.48 | learning rate: 1.179E-04 | global batch size:   256 | lm loss: 3.473159E+00 | grad norm: 0.345 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.621 | TFLOPs: 35.16 |
7:  iteration     5390/   11269 | consumed samples:      1379840 | consumed tokens:   2825912320 | elapsed time per iteration (s): 0.48 | learning rate: 1.176E-04 | global batch size:   256 | lm loss: 3.474450E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.920 | TFLOPs: 34.92 |
7:  iteration     5400/   11269 | consumed samples:      1382400 | consumed tokens:   2831155200 | elapsed time per iteration (s): 0.48 | learning rate: 1.174E-04 | global batch size:   256 | lm loss: 3.469470E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.143 | TFLOPs: 35.19 |
7:  iteration     5410/   11269 | consumed samples:      1384960 | consumed tokens:   2836398080 | elapsed time per iteration (s): 0.48 | learning rate: 1.171E-04 | global batch size:   256 | lm loss: 3.467048E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.935 | TFLOPs: 35.18 |
7:  iteration     5420/   11269 | consumed samples:      1387520 | consumed tokens:   2841640960 | elapsed time per iteration (s): 0.48 | learning rate: 1.169E-04 | global batch size:   256 | lm loss: 3.474428E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.854 | TFLOPs: 34.91 |
7:  iteration     5430/   11269 | consumed samples:      1390080 | consumed tokens:   2846883840 | elapsed time per iteration (s): 0.48 | learning rate: 1.166E-04 | global batch size:   256 | lm loss: 3.475266E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.787 | TFLOPs: 35.11 |
7:  iteration     5440/   11269 | consumed samples:      1392640 | consumed tokens:   2852126720 | elapsed time per iteration (s): 0.48 | learning rate: 1.164E-04 | global batch size:   256 | lm loss: 3.466885E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.429 | TFLOPs: 35.15 |
7:  iteration     5450/   11269 | consumed samples:      1395200 | consumed tokens:   2857369600 | elapsed time per iteration (s): 0.48 | learning rate: 1.161E-04 | global batch size:   256 | lm loss: 3.473208E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.533 | TFLOPs: 35.15 |
7:  iteration     5460/   11269 | consumed samples:      1397760 | consumed tokens:   2862612480 | elapsed time per iteration (s): 0.48 | learning rate: 1.158E-04 | global batch size:   256 | lm loss: 3.479366E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.810 | TFLOPs: 35.17 |
7:  iteration     5470/   11269 | consumed samples:      1400320 | consumed tokens:   2867855360 | elapsed time per iteration (s): 0.48 | learning rate: 1.156E-04 | global batch size:   256 | lm loss: 3.461372E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.594 | TFLOPs: 35.16 |
7:  iteration     5480/   11269 | consumed samples:      1402880 | consumed tokens:   2873098240 | elapsed time per iteration (s): 0.48 | learning rate: 1.153E-04 | global batch size:   256 | lm loss: 3.480242E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.716 | TFLOPs: 35.17 |
7:  iteration     5490/   11269 | consumed samples:      1405440 | consumed tokens:   2878341120 | elapsed time per iteration (s): 0.48 | learning rate: 1.151E-04 | global batch size:   256 | lm loss: 3.466716E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.598 | TFLOPs: 35.16 |
7:  iteration     5500/   11269 | consumed samples:      1408000 | consumed tokens:   2883584000 | elapsed time per iteration (s): 0.48 | learning rate: 1.148E-04 | global batch size:   256 | lm loss: 3.476184E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.806 | TFLOPs: 35.17 |
7:  iteration     5510/   11269 | consumed samples:      1410560 | consumed tokens:   2888826880 | elapsed time per iteration (s): 0.48 | learning rate: 1.146E-04 | global batch size:   256 | lm loss: 3.459838E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.640 | TFLOPs: 34.97 |
7:  iteration     5520/   11269 | consumed samples:      1413120 | consumed tokens:   2894069760 | elapsed time per iteration (s): 0.48 | learning rate: 1.143E-04 | global batch size:   256 | lm loss: 3.462251E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.996 | TFLOPs: 35.18 |
7:  iteration     5530/   11269 | consumed samples:      1415680 | consumed tokens:   2899312640 | elapsed time per iteration (s): 0.48 | learning rate: 1.141E-04 | global batch size:   256 | lm loss: 3.479703E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.248 | TFLOPs: 35.20 |
7:  iteration     5540/   11269 | consumed samples:      1418240 | consumed tokens:   2904555520 | elapsed time per iteration (s): 0.48 | learning rate: 1.138E-04 | global batch size:   256 | lm loss: 3.475929E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.947 | TFLOPs: 34.99 |
7:  iteration     5550/   11269 | consumed samples:      1420800 | consumed tokens:   2909798400 | elapsed time per iteration (s): 0.48 | learning rate: 1.136E-04 | global batch size:   256 | lm loss: 3.464724E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 528.673 | TFLOPs: 34.58 |
7:  iteration     5560/   11269 | consumed samples:      1423360 | consumed tokens:   2915041280 | elapsed time per iteration (s): 0.48 | learning rate: 1.133E-04 | global batch size:   256 | lm loss: 3.460003E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.149 | TFLOPs: 34.74 |
7:  iteration     5570/   11269 | consumed samples:      1425920 | consumed tokens:   2920284160 | elapsed time per iteration (s): 0.48 | learning rate: 1.131E-04 | global batch size:   256 | lm loss: 3.469551E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.017 | TFLOPs: 35.19 |
7:  iteration     5580/   11269 | consumed samples:      1428480 | consumed tokens:   2925527040 | elapsed time per iteration (s): 0.48 | learning rate: 1.128E-04 | global batch size:   256 | lm loss: 3.455764E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.766 | TFLOPs: 35.24 |
7:  iteration     5590/   11269 | consumed samples:      1431040 | consumed tokens:   2930769920 | elapsed time per iteration (s): 0.48 | learning rate: 1.126E-04 | global batch size:   256 | lm loss: 3.465266E+00 | grad norm: 0.342 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.687 | TFLOPs: 35.23 |
7:  iteration     5600/   11269 | consumed samples:      1433600 | consumed tokens:   2936012800 | elapsed time per iteration (s): 0.48 | learning rate: 1.123E-04 | global batch size:   256 | lm loss: 3.453191E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.592 | TFLOPs: 35.03 |
7:  iteration     5610/   11269 | consumed samples:      1436160 | consumed tokens:   2941255680 | elapsed time per iteration (s): 0.48 | learning rate: 1.120E-04 | global batch size:   256 | lm loss: 3.453803E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.054 | TFLOPs: 35.12 |
7:  iteration     5620/   11269 | consumed samples:      1438720 | consumed tokens:   2946498560 | elapsed time per iteration (s): 0.48 | learning rate: 1.118E-04 | global batch size:   256 | lm loss: 3.465922E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.687 | TFLOPs: 35.23 |
7:  iteration     5630/   11269 | consumed samples:      1441280 | consumed tokens:   2951741440 | elapsed time per iteration (s): 0.48 | learning rate: 1.115E-04 | global batch size:   256 | lm loss: 3.453612E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.494 | TFLOPs: 35.22 |
7:  iteration     5640/   11269 | consumed samples:      1443840 | consumed tokens:   2956984320 | elapsed time per iteration (s): 0.48 | learning rate: 1.113E-04 | global batch size:   256 | lm loss: 3.454318E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.383 | TFLOPs: 34.95 |
7:  iteration     5650/   11269 | consumed samples:      1446400 | consumed tokens:   2962227200 | elapsed time per iteration (s): 0.48 | learning rate: 1.110E-04 | global batch size:   256 | lm loss: 3.451409E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.415 | TFLOPs: 35.15 |
7:  iteration     5660/   11269 | consumed samples:      1448960 | consumed tokens:   2967470080 | elapsed time per iteration (s): 0.48 | learning rate: 1.108E-04 | global batch size:   256 | lm loss: 3.467605E+00 | grad norm: 0.343 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.298 | TFLOPs: 35.20 |
7:  iteration     5670/   11269 | consumed samples:      1451520 | consumed tokens:   2972712960 | elapsed time per iteration (s): 0.48 | learning rate: 1.105E-04 | global batch size:   256 | lm loss: 3.443607E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.668 | TFLOPs: 35.23 |
7:  iteration     5680/   11269 | consumed samples:      1454080 | consumed tokens:   2977955840 | elapsed time per iteration (s): 0.48 | learning rate: 1.103E-04 | global batch size:   256 | lm loss: 3.450143E+00 | grad norm: 0.332 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.990 | TFLOPs: 35.05 |
7:  iteration     5690/   11269 | consumed samples:      1456640 | consumed tokens:   2983198720 | elapsed time per iteration (s): 0.48 | learning rate: 1.100E-04 | global batch size:   256 | lm loss: 3.440339E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.518 | TFLOPs: 35.22 |
7:  iteration     5700/   11269 | consumed samples:      1459200 | consumed tokens:   2988441600 | elapsed time per iteration (s): 0.48 | learning rate: 1.098E-04 | global batch size:   256 | lm loss: 3.464775E+00 | grad norm: 0.333 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.040 | TFLOPs: 35.06 |
7:  iteration     5710/   11269 | consumed samples:      1461760 | consumed tokens:   2993684480 | elapsed time per iteration (s): 0.48 | learning rate: 1.095E-04 | global batch size:   256 | lm loss: 3.431411E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.385 | TFLOPs: 34.95 |
7:  iteration     5720/   11269 | consumed samples:      1464320 | consumed tokens:   2998927360 | elapsed time per iteration (s): 0.48 | learning rate: 1.093E-04 | global batch size:   256 | lm loss: 3.442437E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.929 | TFLOPs: 35.25 |
7:  iteration     5730/   11269 | consumed samples:      1466880 | consumed tokens:   3004170240 | elapsed time per iteration (s): 0.47 | learning rate: 1.090E-04 | global batch size:   256 | lm loss: 3.453783E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.239 | TFLOPs: 35.27 |
7:  iteration     5740/   11269 | consumed samples:      1469440 | consumed tokens:   3009413120 | elapsed time per iteration (s): 0.47 | learning rate: 1.088E-04 | global batch size:   256 | lm loss: 3.445419E+00 | grad norm: 0.335 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.125 | TFLOPs: 35.26 |
7:  iteration     5750/   11269 | consumed samples:      1472000 | consumed tokens:   3014656000 | elapsed time per iteration (s): 0.47 | learning rate: 1.085E-04 | global batch size:   256 | lm loss: 3.441238E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.015 | TFLOPs: 35.25 |
7:  iteration     5760/   11269 | consumed samples:      1474560 | consumed tokens:   3019898880 | elapsed time per iteration (s): 0.48 | learning rate: 1.082E-04 | global batch size:   256 | lm loss: 3.454681E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.904 | TFLOPs: 35.24 |
7:  iteration     5770/   11269 | consumed samples:      1477120 | consumed tokens:   3025141760 | elapsed time per iteration (s): 0.48 | learning rate: 1.080E-04 | global batch size:   256 | lm loss: 3.453607E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.597 | TFLOPs: 35.22 |
7:  iteration     5780/   11269 | consumed samples:      1479680 | consumed tokens:   3030384640 | elapsed time per iteration (s): 0.48 | learning rate: 1.077E-04 | global batch size:   256 | lm loss: 3.446737E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.032 | TFLOPs: 34.93 |
7:  iteration     5790/   11269 | consumed samples:      1482240 | consumed tokens:   3035627520 | elapsed time per iteration (s): 0.47 | learning rate: 1.075E-04 | global batch size:   256 | lm loss: 3.444495E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.572 | TFLOPs: 35.29 |
7:  iteration     5800/   11269 | consumed samples:      1484800 | consumed tokens:   3040870400 | elapsed time per iteration (s): 0.47 | learning rate: 1.072E-04 | global batch size:   256 | lm loss: 3.456130E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.244 | TFLOPs: 35.27 |
7:  iteration     5810/   11269 | consumed samples:      1487360 | consumed tokens:   3046113280 | elapsed time per iteration (s): 0.47 | learning rate: 1.070E-04 | global batch size:   256 | lm loss: 3.447393E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.065 | TFLOPs: 35.25 |
7:  iteration     5820/   11269 | consumed samples:      1489920 | consumed tokens:   3051356160 | elapsed time per iteration (s): 0.47 | learning rate: 1.067E-04 | global batch size:   256 | lm loss: 3.450786E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.126 | TFLOPs: 35.26 |
7:  iteration     5830/   11269 | consumed samples:      1492480 | consumed tokens:   3056599040 | elapsed time per iteration (s): 0.48 | learning rate: 1.065E-04 | global batch size:   256 | lm loss: 3.455126E+00 | grad norm: 0.342 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.642 | TFLOPs: 34.90 |
7:  iteration     5840/   11269 | consumed samples:      1495040 | consumed tokens:   3061841920 | elapsed time per iteration (s): 0.47 | learning rate: 1.062E-04 | global batch size:   256 | lm loss: 3.452491E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.169 | TFLOPs: 35.26 |
7:  iteration     5850/   11269 | consumed samples:      1497600 | consumed tokens:   3067084800 | elapsed time per iteration (s): 0.47 | learning rate: 1.060E-04 | global batch size:   256 | lm loss: 3.449771E+00 | grad norm: 0.602 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.091 | TFLOPs: 35.26 |
7:  iteration     5860/   11269 | consumed samples:      1500160 | consumed tokens:   3072327680 | elapsed time per iteration (s): 0.47 | learning rate: 1.057E-04 | global batch size:   256 | lm loss: 3.448683E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.399 | TFLOPs: 35.28 |
7:  iteration     5870/   11269 | consumed samples:      1502720 | consumed tokens:   3077570560 | elapsed time per iteration (s): 0.48 | learning rate: 1.055E-04 | global batch size:   256 | lm loss: 3.435830E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.271 | TFLOPs: 34.94 |
7:  iteration     5880/   11269 | consumed samples:      1505280 | consumed tokens:   3082813440 | elapsed time per iteration (s): 0.48 | learning rate: 1.052E-04 | global batch size:   256 | lm loss: 3.443459E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.036 | TFLOPs: 35.06 |
7:  iteration     5890/   11269 | consumed samples:      1507840 | consumed tokens:   3088056320 | elapsed time per iteration (s): 0.47 | learning rate: 1.050E-04 | global batch size:   256 | lm loss: 3.454259E+00 | grad norm: 0.347 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.713 | TFLOPs: 35.30 |
7:  iteration     5900/   11269 | consumed samples:      1510400 | consumed tokens:   3093299200 | elapsed time per iteration (s): 0.47 | learning rate: 1.047E-04 | global batch size:   256 | lm loss: 3.454796E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.398 | TFLOPs: 35.28 |
7:  iteration     5910/   11269 | consumed samples:      1512960 | consumed tokens:   3098542080 | elapsed time per iteration (s): 0.48 | learning rate: 1.045E-04 | global batch size:   256 | lm loss: 3.434761E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.087 | TFLOPs: 34.99 |
7:  iteration     5920/   11269 | consumed samples:      1515520 | consumed tokens:   3103784960 | elapsed time per iteration (s): 0.47 | learning rate: 1.042E-04 | global batch size:   256 | lm loss: 3.435603E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.501 | TFLOPs: 35.28 |
7:  iteration     5930/   11269 | consumed samples:      1518080 | consumed tokens:   3109027840 | elapsed time per iteration (s): 0.48 | learning rate: 1.039E-04 | global batch size:   256 | lm loss: 3.451527E+00 | grad norm: 0.320 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.560 | TFLOPs: 35.03 |
7:  iteration     5940/   11269 | consumed samples:      1520640 | consumed tokens:   3114270720 | elapsed time per iteration (s): 0.48 | learning rate: 1.037E-04 | global batch size:   256 | lm loss: 3.448270E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.319 | TFLOPs: 35.08 |
7:  iteration     5950/   11269 | consumed samples:      1523200 | consumed tokens:   3119513600 | elapsed time per iteration (s): 0.47 | learning rate: 1.034E-04 | global batch size:   256 | lm loss: 3.436526E+00 | grad norm: 0.378 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.596 | TFLOPs: 35.29 |
7:  iteration     5960/   11269 | consumed samples:      1525760 | consumed tokens:   3124756480 | elapsed time per iteration (s): 0.47 | learning rate: 1.032E-04 | global batch size:   256 | lm loss: 3.436576E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.718 | TFLOPs: 35.30 |
7:  iteration     5970/   11269 | consumed samples:      1528320 | consumed tokens:   3129999360 | elapsed time per iteration (s): 0.47 | learning rate: 1.029E-04 | global batch size:   256 | lm loss: 3.444768E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.069 | TFLOPs: 35.32 |
7:  iteration     5980/   11269 | consumed samples:      1530880 | consumed tokens:   3135242240 | elapsed time per iteration (s): 0.47 | learning rate: 1.027E-04 | global batch size:   256 | lm loss: 3.457648E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.078 | TFLOPs: 35.26 |
7:  iteration     5990/   11269 | consumed samples:      1533440 | consumed tokens:   3140485120 | elapsed time per iteration (s): 0.48 | learning rate: 1.024E-04 | global batch size:   256 | lm loss: 3.452099E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.252 | TFLOPs: 35.07 |
0: [2023-03-15 22:47:24,325] [INFO] [logging.py:68:log_dist] [Rank 0] step=6000, skipped=0, lr=[0.00010217547488893524, 0.00010217547488893524, 0.00010217547488893524], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration     6000/   11269 | consumed samples:      1536000 | consumed tokens:   3145728000 | elapsed time per iteration (s): 0.48 | learning rate: 1.022E-04 | global batch size:   256 | lm loss: 3.433497E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.032 | TFLOPs: 34.99 |
0: steps: 6000 loss: 3.4109 iter time (s): 0.473 samples/sec: 540.759
7: -----------------------------------------------------------------------------------------------
7: validation loss at iteration 6000 | lm loss value: 3.504254E+00 | lm loss PPL: 3.325663E+01 | 
7: -----------------------------------------------------------------------------------------------
0: saving checkpoint at iteration    6000 to checkpoints_280m5b9400m
0: [2023-03-15 22:47:24,503] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step6000 is begin to save!
0: [2023-03-15 22:47:24,506] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_01-model_00-model_states.pt...
0: [2023-03-15 22:47:24,617] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_01-model_00-model_states.pt.
0: [2023-03-15 22:47:24,617] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_03-model_00-model_states.pt...
0: [2023-03-15 22:47:24,642] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_03-model_00-model_states.pt.
0: [2023-03-15 22:47:24,642] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_04-model_00-model_states.pt...
0: [2023-03-15 22:47:24,666] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_04-model_00-model_states.pt.
0: [2023-03-15 22:47:24,667] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_05-model_00-model_states.pt...
0: [2023-03-15 22:47:24,691] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_05-model_00-model_states.pt.
0: [2023-03-15 22:47:24,691] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_06-model_00-model_states.pt...
0: [2023-03-15 22:47:24,715] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_06-model_00-model_states.pt.
0: [2023-03-15 22:47:24,715] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_07-model_00-model_states.pt...
0: [2023-03-15 22:47:24,739] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_07-model_00-model_states.pt.
0: [2023-03-15 22:47:24,740] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_08-model_00-model_states.pt...
0: [2023-03-15 22:47:24,764] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_08-model_00-model_states.pt.
0: [2023-03-15 22:47:24,764] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_09-model_00-model_states.pt...
0: [2023-03-15 22:47:24,788] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_09-model_00-model_states.pt.
0: [2023-03-15 22:47:24,788] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_10-model_00-model_states.pt...
0: [2023-03-15 22:47:24,812] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_10-model_00-model_states.pt.
0: [2023-03-15 22:47:24,812] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_11-model_00-model_states.pt...
0: [2023-03-15 22:47:24,836] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_11-model_00-model_states.pt.
0: [2023-03-15 22:47:24,837] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_12-model_00-model_states.pt...
0: [2023-03-15 22:47:24,861] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_12-model_00-model_states.pt.
0: [2023-03-15 22:47:24,861] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_13-model_00-model_states.pt...
0: [2023-03-15 22:47:24,885] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_13-model_00-model_states.pt.
0: [2023-03-15 22:47:24,885] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_14-model_00-model_states.pt...
0: [2023-03-15 22:47:24,909] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_14-model_00-model_states.pt.
0: [2023-03-15 22:47:24,909] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_15-model_00-model_states.pt...
0: [2023-03-15 22:47:24,933] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_15-model_00-model_states.pt.
0: [2023-03-15 22:47:24,934] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_16-model_00-model_states.pt...
0: [2023-03-15 22:47:24,958] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_16-model_00-model_states.pt.
0: [2023-03-15 22:47:24,958] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_17-model_00-model_states.pt...
0: [2023-03-15 22:47:24,982] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_17-model_00-model_states.pt.
0: [2023-03-15 22:47:24,982] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_18-model_00-model_states.pt...
0: [2023-03-15 22:47:25,006] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_18-model_00-model_states.pt.
0: [2023-03-15 22:47:25,006] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_19-model_00-model_states.pt...
0: [2023-03-15 22:47:25,030] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_19-model_00-model_states.pt.
0: [2023-03-15 22:47:25,031] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_20-model_00-model_states.pt...
0: [2023-03-15 22:47:25,055] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_20-model_00-model_states.pt.
0: [2023-03-15 22:47:25,055] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/layer_22-model_00-model_states.pt...
0: [2023-03-15 22:47:25,056] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/layer_22-model_00-model_states.pt.
0: [2023-03-15 22:47:25,057] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_280m5b9400m/global_step6000/mp_rank_00_model_states.pt
0: [2023-03-15 22:47:25,057] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/mp_rank_00_model_states.pt...
0: [2023-03-15 22:47:25,059] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/mp_rank_00_model_states.pt.
0: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:47:25,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:47:25,132] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:47:25,135] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:47:25,135] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-15 22:47:25,135] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
5: [2023-03-15 22:47:25,136] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:47:25,136] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-15 22:47:25,136] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
4: [2023-03-15 22:47:25,136] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:47:25,136] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-15 22:47:25,136] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
1: [2023-03-15 22:47:25,139] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:47:25,139] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:47:25,140] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-15 22:47:25,140] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-15 22:47:25,140] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
1: [2023-03-15 22:47:25,140] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
1: [2023-03-15 22:47:25,140] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:47:25,140] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-15 22:47:25,140] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
6: [2023-03-15 22:47:25,142] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:47:25,142] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-15 22:47:25,142] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
6: [2023-03-15 22:47:25,143] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:47:25,143] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-15 22:47:25,143] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
6: [2023-03-15 22:47:25,143] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:47:25,144] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-15 22:47:25,144] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
7: [2023-03-15 22:47:25,144] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:47:25,144] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-15 22:47:25,144] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
1: [2023-03-15 22:47:25,144] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:47:25,144] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-15 22:47:25,144] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
4: [2023-03-15 22:47:25,145] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:47:25,145] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:47:25,145] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-15 22:47:25,145] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-15 22:47:25,145] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
4: [2023-03-15 22:47:25,145] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
3: [2023-03-15 22:47:25,145] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:47:25,145] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:47:25,145] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-15 22:47:25,145] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
4: [2023-03-15 22:47:25,145] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:47:25,145] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-15 22:47:25,145] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
5: [2023-03-15 22:47:25,147] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:47:25,147] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-15 22:47:25,147] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
6: [2023-03-15 22:47:25,147] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:47:25,147] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-15 22:47:25,147] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
4: [2023-03-15 22:47:25,147] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:47:25,147] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-15 22:47:25,147] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
1: [2023-03-15 22:47:25,147] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:47:25,148] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-15 22:47:25,148] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
5: [2023-03-15 22:47:25,148] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:47:25,148] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:47:25,148] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
4: [2023-03-15 22:47:25,148] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:47:25,148] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
5: [2023-03-15 22:47:25,148] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
4: [2023-03-15 22:47:25,148] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
4: [2023-03-15 22:47:25,148] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-15 22:47:25,148] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
5: [2023-03-15 22:47:25,148] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:47:25,148] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-15 22:47:25,148] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
1: [2023-03-15 22:47:25,148] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:47:25,148] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
4: [2023-03-15 22:47:25,148] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:47:25,149] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
4: [2023-03-15 22:47:25,149] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-15 22:47:25,149] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
6: [2023-03-15 22:47:25,150] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:47:25,150] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-15 22:47:25,151] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
2: [2023-03-15 22:47:25,143] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:47:25,143] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:47:25,143] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-15 22:47:25,144] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
2: [2023-03-15 22:47:25,144] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-15 22:47:25,144] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
2: [2023-03-15 22:47:25,150] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:47:25,150] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-15 22:47:25,150] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
2: [2023-03-15 22:47:25,151] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:47:25,151] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:47:25,151] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-15 22:47:25,151] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-15 22:47:25,151] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
2: [2023-03-15 22:47:25,151] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
7: [2023-03-15 22:47:25,153] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:47:25,154] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-15 22:47:25,154] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
7: [2023-03-15 22:47:25,154] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:47:25,154] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:47:25,154] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:47:25,154] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-15 22:47:25,154] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:47:25,154] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-15 22:47:25,154] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-15 22:47:25,154] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:47:25,154] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:47:25,154] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
7: [2023-03-15 22:47:25,154] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-15 22:47:25,154] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
7: [2023-03-15 22:47:25,154] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
7: [2023-03-15 22:47:25,154] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-15 22:47:25,154] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-15 22:47:25,154] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
7: [2023-03-15 22:47:25,154] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
7: [2023-03-15 22:47:25,154] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
3: [2023-03-15 22:47:25,145] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-15 22:47:25,145] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
3: [2023-03-15 22:47:25,145] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:47:25,145] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-15 22:47:25,146] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
3: [2023-03-15 22:47:25,145] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:47:25,146] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-15 22:47:25,146] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
3: [2023-03-15 22:47:25,147] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:47:25,147] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-15 22:47:25,147] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
3: [2023-03-15 22:47:25,147] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:47:25,147] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:47:25,147] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:47:25,147] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-15 22:47:25,147] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-15 22:47:25,147] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-15 22:47:25,147] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
3: [2023-03-15 22:47:25,147] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
3: [2023-03-15 22:47:25,147] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
3: [2023-03-15 22:47:25,152] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:47:25,152] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-15 22:47:25,152] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
0: [2023-03-15 22:47:25,162] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:47:25,162] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:47:25,162] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:47:25,162] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:47:25,162] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-15 22:47:25,162] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-15 22:47:25,162] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-15 22:47:25,162] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-15 22:47:25,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
0: [2023-03-15 22:47:25,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
0: [2023-03-15 22:47:25,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
0: [2023-03-15 22:47:25,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
2: [2023-03-15 22:47:25,163] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:47:25,163] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-15 22:47:25,163] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
2: [2023-03-15 22:47:25,163] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:47:25,163] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-15 22:47:25,163] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
2: [2023-03-15 22:47:25,163] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:47:25,163] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-15 22:47:25,163] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
1: [2023-03-15 22:47:25,175] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:47:25,175] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:47:25,175] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-15 22:47:25,175] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-15 22:47:25,175] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
1: [2023-03-15 22:47:25,175] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
6: [2023-03-15 22:47:25,213] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:47:25,213] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-15 22:47:25,213] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
5: [2023-03-15 22:47:25,222] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:47:25,222] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-15 22:47:25,222] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
5: [2023-03-15 22:47:25,223] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:47:25,223] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-15 22:47:25,223] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
0: [2023-03-15 22:47:25,229] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:47:25,229] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:47:25,229] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:47:25,229] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-15 22:47:25,229] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-15 22:47:25,229] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
0: [2023-03-15 22:47:25,229] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-15 22:47:25,229] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
0: [2023-03-15 22:47:25,229] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
0: [2023-03-15 22:47:25,232] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-15 22:47:25,232] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
6: [2023-03-15 22:47:25,243] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:47:25,243] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-15 22:47:25,243] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
6: [2023-03-15 22:47:25,245] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:47:25,245] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step6000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-15 22:47:25,245] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step6000 is ready now!
0:   successfully saved checkpoint at iteration    6000 to checkpoints_280m5b9400m
7: time (ms) | save-checkpoint: 746.36
7:  iteration     6010/   11269 | consumed samples:      1538560 | consumed tokens:   3150970880 | elapsed time per iteration (s): 0.56 | learning rate: 1.019E-04 | global batch size:   256 | lm loss: 3.439256E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 454.751 | TFLOPs: 29.74 |
7:  iteration     6020/   11269 | consumed samples:      1541120 | consumed tokens:   3156213760 | elapsed time per iteration (s): 0.47 | learning rate: 1.017E-04 | global batch size:   256 | lm loss: 3.437464E+00 | grad norm: 0.332 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.282 | TFLOPs: 35.40 |
7:  iteration     6030/   11269 | consumed samples:      1543680 | consumed tokens:   3161456640 | elapsed time per iteration (s): 0.47 | learning rate: 1.014E-04 | global batch size:   256 | lm loss: 3.431055E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.482 | TFLOPs: 35.35 |
7:  iteration     6040/   11269 | consumed samples:      1546240 | consumed tokens:   3166699520 | elapsed time per iteration (s): 0.47 | learning rate: 1.012E-04 | global batch size:   256 | lm loss: 3.445513E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.026 | TFLOPs: 35.32 |
7:  iteration     6050/   11269 | consumed samples:      1548800 | consumed tokens:   3171942400 | elapsed time per iteration (s): 0.48 | learning rate: 1.009E-04 | global batch size:   256 | lm loss: 3.439691E+00 | grad norm: 0.285 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.018 | TFLOPs: 35.06 |
7:  iteration     6060/   11269 | consumed samples:      1551360 | consumed tokens:   3177185280 | elapsed time per iteration (s): 0.47 | learning rate: 1.007E-04 | global batch size:   256 | lm loss: 3.448228E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.984 | TFLOPs: 35.31 |
7:  iteration     6070/   11269 | consumed samples:      1553920 | consumed tokens:   3182428160 | elapsed time per iteration (s): 0.47 | learning rate: 1.004E-04 | global batch size:   256 | lm loss: 3.434833E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.224 | TFLOPs: 35.33 |
7:  iteration     6080/   11269 | consumed samples:      1556480 | consumed tokens:   3187671040 | elapsed time per iteration (s): 0.47 | learning rate: 1.002E-04 | global batch size:   256 | lm loss: 3.422869E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.930 | TFLOPs: 35.31 |
7:  iteration     6090/   11269 | consumed samples:      1559040 | consumed tokens:   3192913920 | elapsed time per iteration (s): 0.47 | learning rate: 9.991E-05 | global batch size:   256 | lm loss: 3.421154E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.307 | TFLOPs: 35.27 |
7:  iteration     6100/   11269 | consumed samples:      1561600 | consumed tokens:   3198156800 | elapsed time per iteration (s): 0.47 | learning rate: 9.965E-05 | global batch size:   256 | lm loss: 3.432378E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.739 | TFLOPs: 35.30 |
7:  iteration     6110/   11269 | consumed samples:      1564160 | consumed tokens:   3203399680 | elapsed time per iteration (s): 0.47 | learning rate: 9.940E-05 | global batch size:   256 | lm loss: 3.437401E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.780 | TFLOPs: 35.30 |
7:  iteration     6120/   11269 | consumed samples:      1566720 | consumed tokens:   3208642560 | elapsed time per iteration (s): 0.47 | learning rate: 9.915E-05 | global batch size:   256 | lm loss: 3.442127E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.950 | TFLOPs: 35.31 |
7:  iteration     6130/   11269 | consumed samples:      1569280 | consumed tokens:   3213885440 | elapsed time per iteration (s): 0.47 | learning rate: 9.890E-05 | global batch size:   256 | lm loss: 3.432441E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.644 | TFLOPs: 35.29 |
7:  iteration     6140/   11269 | consumed samples:      1571840 | consumed tokens:   3219128320 | elapsed time per iteration (s): 0.47 | learning rate: 9.865E-05 | global batch size:   256 | lm loss: 3.430826E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.220 | TFLOPs: 35.26 |
7:  iteration     6150/   11269 | consumed samples:      1574400 | consumed tokens:   3224371200 | elapsed time per iteration (s): 0.47 | learning rate: 9.840E-05 | global batch size:   256 | lm loss: 3.439803E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.616 | TFLOPs: 35.29 |
7:  iteration     6160/   11269 | consumed samples:      1576960 | consumed tokens:   3229614080 | elapsed time per iteration (s): 0.48 | learning rate: 9.815E-05 | global batch size:   256 | lm loss: 3.423114E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.472 | TFLOPs: 34.95 |
7:  iteration     6170/   11269 | consumed samples:      1579520 | consumed tokens:   3234856960 | elapsed time per iteration (s): 0.48 | learning rate: 9.789E-05 | global batch size:   256 | lm loss: 3.425268E+00 | grad norm: 0.327 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.462 | TFLOPs: 35.08 |
7:  iteration     6180/   11269 | consumed samples:      1582080 | consumed tokens:   3240099840 | elapsed time per iteration (s): 0.47 | learning rate: 9.764E-05 | global batch size:   256 | lm loss: 3.436739E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.529 | TFLOPs: 35.29 |
7:  iteration     6190/   11269 | consumed samples:      1584640 | consumed tokens:   3245342720 | elapsed time per iteration (s): 0.47 | learning rate: 9.739E-05 | global batch size:   256 | lm loss: 3.421218E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.565 | TFLOPs: 35.29 |
7:  iteration     6200/   11269 | consumed samples:      1587200 | consumed tokens:   3250585600 | elapsed time per iteration (s): 0.47 | learning rate: 9.714E-05 | global batch size:   256 | lm loss: 3.430359E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.583 | TFLOPs: 35.29 |
7:  iteration     6210/   11269 | consumed samples:      1589760 | consumed tokens:   3255828480 | elapsed time per iteration (s): 0.47 | learning rate: 9.689E-05 | global batch size:   256 | lm loss: 3.420074E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.939 | TFLOPs: 35.31 |
7:  iteration     6220/   11269 | consumed samples:      1592320 | consumed tokens:   3261071360 | elapsed time per iteration (s): 0.47 | learning rate: 9.664E-05 | global batch size:   256 | lm loss: 3.421231E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.469 | TFLOPs: 35.28 |
7:  iteration     6230/   11269 | consumed samples:      1594880 | consumed tokens:   3266314240 | elapsed time per iteration (s): 0.47 | learning rate: 9.639E-05 | global batch size:   256 | lm loss: 3.412927E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.515 | TFLOPs: 35.28 |
7:  iteration     6240/   11269 | consumed samples:      1597440 | consumed tokens:   3271557120 | elapsed time per iteration (s): 0.47 | learning rate: 9.614E-05 | global batch size:   256 | lm loss: 3.419145E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.724 | TFLOPs: 35.30 |
7:  iteration     6250/   11269 | consumed samples:      1600000 | consumed tokens:   3276800000 | elapsed time per iteration (s): 0.47 | learning rate: 9.589E-05 | global batch size:   256 | lm loss: 3.421911E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.186 | TFLOPs: 35.26 |
7:  iteration     6260/   11269 | consumed samples:      1602560 | consumed tokens:   3282042880 | elapsed time per iteration (s): 0.47 | learning rate: 9.564E-05 | global batch size:   256 | lm loss: 3.419669E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.373 | TFLOPs: 35.27 |
7:  iteration     6270/   11269 | consumed samples:      1605120 | consumed tokens:   3287285760 | elapsed time per iteration (s): 0.47 | learning rate: 9.539E-05 | global batch size:   256 | lm loss: 3.421324E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.177 | TFLOPs: 35.26 |
7:  iteration     6280/   11269 | consumed samples:      1607680 | consumed tokens:   3292528640 | elapsed time per iteration (s): 0.47 | learning rate: 9.514E-05 | global batch size:   256 | lm loss: 3.427113E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.297 | TFLOPs: 35.27 |
7:  iteration     6290/   11269 | consumed samples:      1610240 | consumed tokens:   3297771520 | elapsed time per iteration (s): 0.47 | learning rate: 9.489E-05 | global batch size:   256 | lm loss: 3.412718E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.066 | TFLOPs: 35.25 |
7:  iteration     6300/   11269 | consumed samples:      1612800 | consumed tokens:   3303014400 | elapsed time per iteration (s): 0.47 | learning rate: 9.464E-05 | global batch size:   256 | lm loss: 3.410367E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.156 | TFLOPs: 35.26 |
7:  iteration     6310/   11269 | consumed samples:      1615360 | consumed tokens:   3308257280 | elapsed time per iteration (s): 0.47 | learning rate: 9.439E-05 | global batch size:   256 | lm loss: 3.413623E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.434 | TFLOPs: 35.28 |
7:  iteration     6320/   11269 | consumed samples:      1617920 | consumed tokens:   3313500160 | elapsed time per iteration (s): 0.47 | learning rate: 9.414E-05 | global batch size:   256 | lm loss: 3.408359E+00 | grad norm: 0.278 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.210 | TFLOPs: 35.26 |
7:  iteration     6330/   11269 | consumed samples:      1620480 | consumed tokens:   3318743040 | elapsed time per iteration (s): 0.47 | learning rate: 9.389E-05 | global batch size:   256 | lm loss: 3.418362E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.275 | TFLOPs: 35.27 |
7:  iteration     6340/   11269 | consumed samples:      1623040 | consumed tokens:   3323985920 | elapsed time per iteration (s): 0.47 | learning rate: 9.364E-05 | global batch size:   256 | lm loss: 3.407975E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.542 | TFLOPs: 35.29 |
7:  iteration     6350/   11269 | consumed samples:      1625600 | consumed tokens:   3329228800 | elapsed time per iteration (s): 0.47 | learning rate: 9.339E-05 | global batch size:   256 | lm loss: 3.423827E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.067 | TFLOPs: 35.25 |
7:  iteration     6360/   11269 | consumed samples:      1628160 | consumed tokens:   3334471680 | elapsed time per iteration (s): 0.47 | learning rate: 9.314E-05 | global batch size:   256 | lm loss: 3.415406E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.049 | TFLOPs: 35.25 |
7:  iteration     6370/   11269 | consumed samples:      1630720 | consumed tokens:   3339714560 | elapsed time per iteration (s): 0.48 | learning rate: 9.289E-05 | global batch size:   256 | lm loss: 3.410185E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.432 | TFLOPs: 35.08 |
7:  iteration     6380/   11269 | consumed samples:      1633280 | consumed tokens:   3344957440 | elapsed time per iteration (s): 0.47 | learning rate: 9.264E-05 | global batch size:   256 | lm loss: 3.414592E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.281 | TFLOPs: 35.27 |
7:  iteration     6390/   11269 | consumed samples:      1635840 | consumed tokens:   3350200320 | elapsed time per iteration (s): 0.47 | learning rate: 9.240E-05 | global batch size:   256 | lm loss: 3.412292E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.192 | TFLOPs: 35.26 |
7:  iteration     6400/   11269 | consumed samples:      1638400 | consumed tokens:   3355443200 | elapsed time per iteration (s): 0.47 | learning rate: 9.215E-05 | global batch size:   256 | lm loss: 3.408649E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.198 | TFLOPs: 35.26 |
7:  iteration     6410/   11269 | consumed samples:      1640960 | consumed tokens:   3360686080 | elapsed time per iteration (s): 0.48 | learning rate: 9.190E-05 | global batch size:   256 | lm loss: 3.411223E+00 | grad norm: 0.343 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.934 | TFLOPs: 35.25 |
7:  iteration     6420/   11269 | consumed samples:      1643520 | consumed tokens:   3365928960 | elapsed time per iteration (s): 0.47 | learning rate: 9.165E-05 | global batch size:   256 | lm loss: 3.400182E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.471 | TFLOPs: 35.28 |
7:  iteration     6430/   11269 | consumed samples:      1646080 | consumed tokens:   3371171840 | elapsed time per iteration (s): 0.47 | learning rate: 9.140E-05 | global batch size:   256 | lm loss: 3.415942E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.074 | TFLOPs: 35.26 |
7:  iteration     6440/   11269 | consumed samples:      1648640 | consumed tokens:   3376414720 | elapsed time per iteration (s): 0.48 | learning rate: 9.115E-05 | global batch size:   256 | lm loss: 3.406147E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.709 | TFLOPs: 35.23 |
7:  iteration     6450/   11269 | consumed samples:      1651200 | consumed tokens:   3381657600 | elapsed time per iteration (s): 0.47 | learning rate: 9.091E-05 | global batch size:   256 | lm loss: 3.424207E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.110 | TFLOPs: 35.26 |
7:  iteration     6460/   11269 | consumed samples:      1653760 | consumed tokens:   3386900480 | elapsed time per iteration (s): 0.48 | learning rate: 9.066E-05 | global batch size:   256 | lm loss: 3.403555E+00 | grad norm: 0.331 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.787 | TFLOPs: 35.24 |
7:  iteration     6470/   11269 | consumed samples:      1656320 | consumed tokens:   3392143360 | elapsed time per iteration (s): 0.47 | learning rate: 9.041E-05 | global batch size:   256 | lm loss: 3.416362E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.077 | TFLOPs: 35.26 |
7:  iteration     6480/   11269 | consumed samples:      1658880 | consumed tokens:   3397386240 | elapsed time per iteration (s): 0.48 | learning rate: 9.016E-05 | global batch size:   256 | lm loss: 3.429414E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.804 | TFLOPs: 35.24 |
7:  iteration     6490/   11269 | consumed samples:      1661440 | consumed tokens:   3402629120 | elapsed time per iteration (s): 0.47 | learning rate: 8.992E-05 | global batch size:   256 | lm loss: 3.417876E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.108 | TFLOPs: 35.26 |
7:  iteration     6500/   11269 | consumed samples:      1664000 | consumed tokens:   3407872000 | elapsed time per iteration (s): 0.47 | learning rate: 8.967E-05 | global batch size:   256 | lm loss: 3.417070E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.049 | TFLOPs: 35.25 |
7:  iteration     6510/   11269 | consumed samples:      1666560 | consumed tokens:   3413114880 | elapsed time per iteration (s): 0.47 | learning rate: 8.942E-05 | global batch size:   256 | lm loss: 3.417630E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.971 | TFLOPs: 35.25 |
7:  iteration     6520/   11269 | consumed samples:      1669120 | consumed tokens:   3418357760 | elapsed time per iteration (s): 0.47 | learning rate: 8.918E-05 | global batch size:   256 | lm loss: 3.418918E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.326 | TFLOPs: 35.27 |
7:  iteration     6530/   11269 | consumed samples:      1671680 | consumed tokens:   3423600640 | elapsed time per iteration (s): 0.48 | learning rate: 8.893E-05 | global batch size:   256 | lm loss: 3.408022E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.516 | TFLOPs: 35.15 |
7:  iteration     6540/   11269 | consumed samples:      1674240 | consumed tokens:   3428843520 | elapsed time per iteration (s): 0.47 | learning rate: 8.868E-05 | global batch size:   256 | lm loss: 3.416422E+00 | grad norm: 0.343 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.157 | TFLOPs: 35.26 |
7:  iteration     6550/   11269 | consumed samples:      1676800 | consumed tokens:   3434086400 | elapsed time per iteration (s): 0.47 | learning rate: 8.844E-05 | global batch size:   256 | lm loss: 3.417532E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.297 | TFLOPs: 35.27 |
7:  iteration     6560/   11269 | consumed samples:      1679360 | consumed tokens:   3439329280 | elapsed time per iteration (s): 0.47 | learning rate: 8.819E-05 | global batch size:   256 | lm loss: 3.404197E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.196 | TFLOPs: 35.26 |
7:  iteration     6570/   11269 | consumed samples:      1681920 | consumed tokens:   3444572160 | elapsed time per iteration (s): 0.47 | learning rate: 8.795E-05 | global batch size:   256 | lm loss: 3.408557E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.158 | TFLOPs: 35.26 |
7:  iteration     6580/   11269 | consumed samples:      1684480 | consumed tokens:   3449815040 | elapsed time per iteration (s): 0.47 | learning rate: 8.770E-05 | global batch size:   256 | lm loss: 3.408694E+00 | grad norm: 0.327 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.316 | TFLOPs: 35.27 |
7:  iteration     6590/   11269 | consumed samples:      1687040 | consumed tokens:   3455057920 | elapsed time per iteration (s): 0.47 | learning rate: 8.746E-05 | global batch size:   256 | lm loss: 3.406874E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.227 | TFLOPs: 35.27 |
7:  iteration     6600/   11269 | consumed samples:      1689600 | consumed tokens:   3460300800 | elapsed time per iteration (s): 0.47 | learning rate: 8.721E-05 | global batch size:   256 | lm loss: 3.417134E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.497 | TFLOPs: 35.28 |
7:  iteration     6610/   11269 | consumed samples:      1692160 | consumed tokens:   3465543680 | elapsed time per iteration (s): 0.47 | learning rate: 8.697E-05 | global batch size:   256 | lm loss: 3.403965E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.462 | TFLOPs: 35.28 |
7:  iteration     6620/   11269 | consumed samples:      1694720 | consumed tokens:   3470786560 | elapsed time per iteration (s): 0.47 | learning rate: 8.672E-05 | global batch size:   256 | lm loss: 3.397976E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.556 | TFLOPs: 35.29 |
7:  iteration     6630/   11269 | consumed samples:      1697280 | consumed tokens:   3476029440 | elapsed time per iteration (s): 0.47 | learning rate: 8.648E-05 | global batch size:   256 | lm loss: 3.395756E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.538 | TFLOPs: 35.29 |
7:  iteration     6640/   11269 | consumed samples:      1699840 | consumed tokens:   3481272320 | elapsed time per iteration (s): 0.47 | learning rate: 8.623E-05 | global batch size:   256 | lm loss: 3.411610E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.573 | TFLOPs: 35.29 |
7:  iteration     6650/   11269 | consumed samples:      1702400 | consumed tokens:   3486515200 | elapsed time per iteration (s): 0.48 | learning rate: 8.599E-05 | global batch size:   256 | lm loss: 3.403269E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.932 | TFLOPs: 35.25 |
7:  iteration     6660/   11269 | consumed samples:      1704960 | consumed tokens:   3491758080 | elapsed time per iteration (s): 0.47 | learning rate: 8.574E-05 | global batch size:   256 | lm loss: 3.404846E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.199 | TFLOPs: 35.26 |
7:  iteration     6670/   11269 | consumed samples:      1707520 | consumed tokens:   3497000960 | elapsed time per iteration (s): 0.48 | learning rate: 8.550E-05 | global batch size:   256 | lm loss: 3.383835E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.923 | TFLOPs: 35.25 |
7:  iteration     6680/   11269 | consumed samples:      1710080 | consumed tokens:   3502243840 | elapsed time per iteration (s): 0.47 | learning rate: 8.525E-05 | global batch size:   256 | lm loss: 3.395335E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.225 | TFLOPs: 35.27 |
7:  iteration     6690/   11269 | consumed samples:      1712640 | consumed tokens:   3507486720 | elapsed time per iteration (s): 0.48 | learning rate: 8.501E-05 | global batch size:   256 | lm loss: 3.408002E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.859 | TFLOPs: 35.24 |
7:  iteration     6700/   11269 | consumed samples:      1715200 | consumed tokens:   3512729600 | elapsed time per iteration (s): 0.47 | learning rate: 8.477E-05 | global batch size:   256 | lm loss: 3.402923E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.193 | TFLOPs: 35.26 |
7:  iteration     6710/   11269 | consumed samples:      1717760 | consumed tokens:   3517972480 | elapsed time per iteration (s): 0.48 | learning rate: 8.452E-05 | global batch size:   256 | lm loss: 3.408067E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.335 | TFLOPs: 35.08 |
7:  iteration     6720/   11269 | consumed samples:      1720320 | consumed tokens:   3523215360 | elapsed time per iteration (s): 0.47 | learning rate: 8.428E-05 | global batch size:   256 | lm loss: 3.405670E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.682 | TFLOPs: 35.30 |
7:  iteration     6730/   11269 | consumed samples:      1722880 | consumed tokens:   3528458240 | elapsed time per iteration (s): 0.47 | learning rate: 8.404E-05 | global batch size:   256 | lm loss: 3.415652E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.926 | TFLOPs: 35.31 |
7:  iteration     6740/   11269 | consumed samples:      1725440 | consumed tokens:   3533701120 | elapsed time per iteration (s): 0.47 | learning rate: 8.380E-05 | global batch size:   256 | lm loss: 3.397996E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.795 | TFLOPs: 35.30 |
7:  iteration     6750/   11269 | consumed samples:      1728000 | consumed tokens:   3538944000 | elapsed time per iteration (s): 0.47 | learning rate: 8.355E-05 | global batch size:   256 | lm loss: 3.403588E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.599 | TFLOPs: 35.29 |
7:  iteration     6760/   11269 | consumed samples:      1730560 | consumed tokens:   3544186880 | elapsed time per iteration (s): 0.47 | learning rate: 8.331E-05 | global batch size:   256 | lm loss: 3.405156E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.084 | TFLOPs: 35.26 |
7:  iteration     6770/   11269 | consumed samples:      1733120 | consumed tokens:   3549429760 | elapsed time per iteration (s): 0.48 | learning rate: 8.307E-05 | global batch size:   256 | lm loss: 3.398749E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.548 | TFLOPs: 35.16 |
7:  iteration     6780/   11269 | consumed samples:      1735680 | consumed tokens:   3554672640 | elapsed time per iteration (s): 0.47 | learning rate: 8.283E-05 | global batch size:   256 | lm loss: 3.403761E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.574 | TFLOPs: 35.29 |
7:  iteration     6790/   11269 | consumed samples:      1738240 | consumed tokens:   3559915520 | elapsed time per iteration (s): 0.47 | learning rate: 8.259E-05 | global batch size:   256 | lm loss: 3.381525E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.084 | TFLOPs: 35.26 |
7:  iteration     6800/   11269 | consumed samples:      1740800 | consumed tokens:   3565158400 | elapsed time per iteration (s): 0.47 | learning rate: 8.235E-05 | global batch size:   256 | lm loss: 3.406296E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.181 | TFLOPs: 35.26 |
7:  iteration     6810/   11269 | consumed samples:      1743360 | consumed tokens:   3570401280 | elapsed time per iteration (s): 0.47 | learning rate: 8.210E-05 | global batch size:   256 | lm loss: 3.407673E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.315 | TFLOPs: 35.27 |
7:  iteration     6820/   11269 | consumed samples:      1745920 | consumed tokens:   3575644160 | elapsed time per iteration (s): 0.47 | learning rate: 8.186E-05 | global batch size:   256 | lm loss: 3.393691E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.468 | TFLOPs: 35.28 |
7:  iteration     6830/   11269 | consumed samples:      1748480 | consumed tokens:   3580887040 | elapsed time per iteration (s): 0.47 | learning rate: 8.162E-05 | global batch size:   256 | lm loss: 3.383891E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.050 | TFLOPs: 35.25 |
7:  iteration     6840/   11269 | consumed samples:      1751040 | consumed tokens:   3586129920 | elapsed time per iteration (s): 0.47 | learning rate: 8.138E-05 | global batch size:   256 | lm loss: 3.396439E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.372 | TFLOPs: 35.27 |
7:  iteration     6850/   11269 | consumed samples:      1753600 | consumed tokens:   3591372800 | elapsed time per iteration (s): 0.47 | learning rate: 8.114E-05 | global batch size:   256 | lm loss: 3.389233E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.027 | TFLOPs: 35.25 |
7:  iteration     6860/   11269 | consumed samples:      1756160 | consumed tokens:   3596615680 | elapsed time per iteration (s): 0.47 | learning rate: 8.090E-05 | global batch size:   256 | lm loss: 3.399974E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.366 | TFLOPs: 35.27 |
7:  iteration     6870/   11269 | consumed samples:      1758720 | consumed tokens:   3601858560 | elapsed time per iteration (s): 0.47 | learning rate: 8.066E-05 | global batch size:   256 | lm loss: 3.389627E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.204 | TFLOPs: 35.26 |
7:  iteration     6880/   11269 | consumed samples:      1761280 | consumed tokens:   3607101440 | elapsed time per iteration (s): 0.48 | learning rate: 8.042E-05 | global batch size:   256 | lm loss: 3.386557E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.780 | TFLOPs: 35.24 |
7:  iteration     6890/   11269 | consumed samples:      1763840 | consumed tokens:   3612344320 | elapsed time per iteration (s): 0.47 | learning rate: 8.018E-05 | global batch size:   256 | lm loss: 3.392017E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.282 | TFLOPs: 35.27 |
7:  iteration     6900/   11269 | consumed samples:      1766400 | consumed tokens:   3617587200 | elapsed time per iteration (s): 0.47 | learning rate: 7.994E-05 | global batch size:   256 | lm loss: 3.392745E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.272 | TFLOPs: 35.27 |
7:  iteration     6910/   11269 | consumed samples:      1768960 | consumed tokens:   3622830080 | elapsed time per iteration (s): 0.47 | learning rate: 7.971E-05 | global batch size:   256 | lm loss: 3.403628E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.099 | TFLOPs: 35.26 |
7:  iteration     6920/   11269 | consumed samples:      1771520 | consumed tokens:   3628072960 | elapsed time per iteration (s): 0.48 | learning rate: 7.947E-05 | global batch size:   256 | lm loss: 3.389312E+00 | grad norm: 0.326 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.298 | TFLOPs: 34.88 |
7:  iteration     6930/   11269 | consumed samples:      1774080 | consumed tokens:   3633315840 | elapsed time per iteration (s): 0.47 | learning rate: 7.923E-05 | global batch size:   256 | lm loss: 3.385606E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.119 | TFLOPs: 35.26 |
7:  iteration     6940/   11269 | consumed samples:      1776640 | consumed tokens:   3638558720 | elapsed time per iteration (s): 0.47 | learning rate: 7.899E-05 | global batch size:   256 | lm loss: 3.384464E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.133 | TFLOPs: 35.26 |
7:  iteration     6950/   11269 | consumed samples:      1779200 | consumed tokens:   3643801600 | elapsed time per iteration (s): 0.47 | learning rate: 7.875E-05 | global batch size:   256 | lm loss: 3.377762E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.326 | TFLOPs: 35.27 |
7:  iteration     6960/   11269 | consumed samples:      1781760 | consumed tokens:   3649044480 | elapsed time per iteration (s): 0.47 | learning rate: 7.852E-05 | global batch size:   256 | lm loss: 3.401046E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.147 | TFLOPs: 35.26 |
7:  iteration     6970/   11269 | consumed samples:      1784320 | consumed tokens:   3654287360 | elapsed time per iteration (s): 0.47 | learning rate: 7.828E-05 | global batch size:   256 | lm loss: 3.387273E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.282 | TFLOPs: 35.27 |
7:  iteration     6980/   11269 | consumed samples:      1786880 | consumed tokens:   3659530240 | elapsed time per iteration (s): 0.47 | learning rate: 7.804E-05 | global batch size:   256 | lm loss: 3.377106E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.109 | TFLOPs: 35.26 |
7:  iteration     6990/   11269 | consumed samples:      1789440 | consumed tokens:   3664773120 | elapsed time per iteration (s): 0.47 | learning rate: 7.780E-05 | global batch size:   256 | lm loss: 3.386224E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.505 | TFLOPs: 35.28 |
7:  iteration     7000/   11269 | consumed samples:      1792000 | consumed tokens:   3670016000 | elapsed time per iteration (s): 0.47 | learning rate: 7.757E-05 | global batch size:   256 | lm loss: 3.387676E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.646 | TFLOPs: 35.29 |
7: -----------------------------------------------------------------------------------------------
7: validation loss at iteration 7000 | lm loss value: 3.488997E+00 | lm loss PPL: 3.275309E+01 | 
7: -----------------------------------------------------------------------------------------------
0: saving checkpoint at iteration    7000 to checkpoints_280m5b9400m
0: [2023-03-15 22:55:20,240] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step7000 is begin to save!
0: [2023-03-15 22:55:20,243] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_01-model_00-model_states.pt...
0: [2023-03-15 22:55:20,356] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_01-model_00-model_states.pt.
0: [2023-03-15 22:55:20,357] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_03-model_00-model_states.pt...
0: [2023-03-15 22:55:20,382] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_03-model_00-model_states.pt.
0: [2023-03-15 22:55:20,382] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_04-model_00-model_states.pt...
0: [2023-03-15 22:55:20,406] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_04-model_00-model_states.pt.
0: [2023-03-15 22:55:20,406] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_05-model_00-model_states.pt...
0: [2023-03-15 22:55:20,430] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_05-model_00-model_states.pt.
0: [2023-03-15 22:55:20,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_06-model_00-model_states.pt...
0: [2023-03-15 22:55:20,455] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_06-model_00-model_states.pt.
0: [2023-03-15 22:55:20,455] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_07-model_00-model_states.pt...
0: [2023-03-15 22:55:20,479] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_07-model_00-model_states.pt.
0: [2023-03-15 22:55:20,479] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_08-model_00-model_states.pt...
0: [2023-03-15 22:55:20,503] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_08-model_00-model_states.pt.
0: [2023-03-15 22:55:20,504] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_09-model_00-model_states.pt...
0: [2023-03-15 22:55:20,528] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_09-model_00-model_states.pt.
0: [2023-03-15 22:55:20,528] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_10-model_00-model_states.pt...
0: [2023-03-15 22:55:20,553] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_10-model_00-model_states.pt.
0: [2023-03-15 22:55:20,553] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_11-model_00-model_states.pt...
0: [2023-03-15 22:55:20,577] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_11-model_00-model_states.pt.
0: [2023-03-15 22:55:20,577] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_12-model_00-model_states.pt...
0: [2023-03-15 22:55:20,602] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_12-model_00-model_states.pt.
0: [2023-03-15 22:55:20,602] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_13-model_00-model_states.pt...
0: [2023-03-15 22:55:20,626] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_13-model_00-model_states.pt.
0: [2023-03-15 22:55:20,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_14-model_00-model_states.pt...
0: [2023-03-15 22:55:20,650] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_14-model_00-model_states.pt.
0: [2023-03-15 22:55:20,651] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_15-model_00-model_states.pt...
0: [2023-03-15 22:55:20,675] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_15-model_00-model_states.pt.
0: [2023-03-15 22:55:20,675] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_16-model_00-model_states.pt...
0: [2023-03-15 22:55:20,699] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_16-model_00-model_states.pt.
0: [2023-03-15 22:55:20,699] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_17-model_00-model_states.pt...
0: [2023-03-15 22:55:20,723] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_17-model_00-model_states.pt.
0: [2023-03-15 22:55:20,723] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_18-model_00-model_states.pt...
0: [2023-03-15 22:55:20,747] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_18-model_00-model_states.pt.
0: [2023-03-15 22:55:20,748] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_19-model_00-model_states.pt...
0: [2023-03-15 22:55:20,772] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_19-model_00-model_states.pt.
0: [2023-03-15 22:55:20,772] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_20-model_00-model_states.pt...
0: [2023-03-15 22:55:20,796] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_20-model_00-model_states.pt.
0: [2023-03-15 22:55:20,796] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/layer_22-model_00-model_states.pt...
0: [2023-03-15 22:55:20,797] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/layer_22-model_00-model_states.pt.
0: [2023-03-15 22:55:20,798] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_280m5b9400m/global_step7000/mp_rank_00_model_states.pt
0: [2023-03-15 22:55:20,798] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/mp_rank_00_model_states.pt...
0: [2023-03-15 22:55:20,801] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/mp_rank_00_model_states.pt.
0: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
2: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
6: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
7: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:55:20,818] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
0: [2023-03-15 22:55:20,876] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:55:20,876] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:55:20,876] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:55:20,876] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-15 22:55:20,876] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
5: [2023-03-15 22:55:20,877] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:55:20,877] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:55:20,877] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-15 22:55:20,877] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
4: [2023-03-15 22:55:20,877] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:55:20,878] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-15 22:55:20,878] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
6: [2023-03-15 22:55:20,879] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:55:20,879] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-15 22:55:20,879] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
6: [2023-03-15 22:55:20,879] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:55:20,879] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-15 22:55:20,880] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
1: [2023-03-15 22:55:20,881] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:55:20,881] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:55:20,881] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-15 22:55:20,881] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-15 22:55:20,881] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
1: [2023-03-15 22:55:20,881] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
1: [2023-03-15 22:55:20,881] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:55:20,881] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-15 22:55:20,881] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
6: [2023-03-15 22:55:20,883] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:55:20,883] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-15 22:55:20,883] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
2: [2023-03-15 22:55:20,884] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:55:20,884] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:55:20,884] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-15 22:55:20,884] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-15 22:55:20,884] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
2: [2023-03-15 22:55:20,884] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
4: [2023-03-15 22:55:20,884] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:55:20,884] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-15 22:55:20,884] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
4: [2023-03-15 22:55:20,884] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:55:20,884] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-15 22:55:20,885] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
4: [2023-03-15 22:55:20,885] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:55:20,885] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-15 22:55:20,885] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
7: [2023-03-15 22:55:20,886] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:55:20,886] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-15 22:55:20,886] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:55:20,886] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
7: [2023-03-15 22:55:20,886] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-15 22:55:20,886] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
7: [2023-03-15 22:55:20,886] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:55:20,887] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-15 22:55:20,887] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
7: [2023-03-15 22:55:20,887] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:55:20,887] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-15 22:55:20,887] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
1: [2023-03-15 22:55:20,887] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:55:20,887] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:55:20,887] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
7: [2023-03-15 22:55:20,887] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:55:20,887] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
7: [2023-03-15 22:55:20,887] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
1: [2023-03-15 22:55:20,887] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
2: [2023-03-15 22:55:20,887] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:55:20,887] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
1: [2023-03-15 22:55:20,887] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
2: [2023-03-15 22:55:20,887] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:55:20,887] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-15 22:55:20,887] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:55:20,887] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-15 22:55:20,887] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
1: [2023-03-15 22:55:20,888] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-15 22:55:20,888] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
2: [2023-03-15 22:55:20,888] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
2: [2023-03-15 22:55:20,888] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
1: [2023-03-15 22:55:20,888] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
1: [2023-03-15 22:55:20,888] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
6: [2023-03-15 22:55:20,889] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:55:20,889] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-15 22:55:20,889] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
2: [2023-03-15 22:55:20,889] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:55:20,889] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-15 22:55:20,889] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
2: [2023-03-15 22:55:20,889] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:55:20,889] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-15 22:55:20,889] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
2: [2023-03-15 22:55:20,890] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-15 22:55:20,890] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-15 22:55:20,890] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
6: [2023-03-15 22:55:20,890] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:55:20,890] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:55:20,890] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-15 22:55:20,890] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-15 22:55:20,890] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
6: [2023-03-15 22:55:20,890] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
6: [2023-03-15 22:55:20,890] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:55:20,890] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-15 22:55:20,890] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-15 22:55:20,890] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-15 22:55:20,890] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
6: [2023-03-15 22:55:20,890] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
4: [2023-03-15 22:55:20,892] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:55:20,893] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-15 22:55:20,893] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
4: [2023-03-15 22:55:20,893] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:55:20,893] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-15 22:55:20,893] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
4: [2023-03-15 22:55:20,893] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:55:20,893] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-15 22:55:20,894] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
0: [2023-03-15 22:55:20,895] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:55:20,895] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:55:20,895] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:55:20,895] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-15 22:55:20,895] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-15 22:55:20,895] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
0: [2023-03-15 22:55:20,895] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
0: [2023-03-15 22:55:20,895] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-15 22:55:20,895] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:55:20,895] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
0: [2023-03-15 22:55:20,895] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-15 22:55:20,895] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
0: [2023-03-15 22:55:20,897] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:55:20,897] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-15 22:55:20,897] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
0: [2023-03-15 22:55:20,897] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:55:20,897] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-15 22:55:20,898] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
0: [2023-03-15 22:55:20,898] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-15 22:55:20,898] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-15 22:55:20,898] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
5: [2023-03-15 22:55:20,877] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-15 22:55:20,877] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
5: [2023-03-15 22:55:20,885] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:55:20,885] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:55:20,885] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-15 22:55:20,885] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-15 22:55:20,885] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
5: [2023-03-15 22:55:20,885] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
5: [2023-03-15 22:55:20,886] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:55:20,886] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:55:20,886] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-15 22:55:20,886] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-15 22:55:20,886] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
5: [2023-03-15 22:55:20,886] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:55:20,886] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
5: [2023-03-15 22:55:20,886] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-15 22:55:20,886] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
7: [2023-03-15 22:55:20,899] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:55:20,899] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-15 22:55:20,899] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
7: [2023-03-15 22:55:20,899] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:55:20,899] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-15 22:55:20,899] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
7: [2023-03-15 22:55:20,900] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-15 22:55:20,900] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-15 22:55:20,900] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
3: [2023-03-15 22:55:20,876] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
4: [2023-03-15 22:55:20,901] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:55:20,876] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
3: [2023-03-15 22:55:20,879] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
4: [2023-03-15 22:55:20,901] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
3: [2023-03-15 22:55:20,879] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
4: [2023-03-15 22:55:20,901] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
3: [2023-03-15 22:55:20,879] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
3: [2023-03-15 22:55:20,881] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:55:20,881] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-15 22:55:20,881] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
3: [2023-03-15 22:55:20,884] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:55:20,884] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-15 22:55:20,884] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
3: [2023-03-15 22:55:20,884] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:55:20,885] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-15 22:55:20,885] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
3: [2023-03-15 22:55:20,885] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:55:20,885] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-15 22:55:20,885] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
3: [2023-03-15 22:55:20,887] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:55:20,887] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-15 22:55:20,887] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
3: [2023-03-15 22:55:20,888] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-15 22:55:20,888] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-15 22:55:20,888] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
0: [2023-03-15 22:55:20,959] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-15 22:55:20,959] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
5: [2023-03-15 22:55:20,977] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:55:20,978] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-15 22:55:20,978] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
5: [2023-03-15 22:55:20,984] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-15 22:55:20,984] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step7000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-15 22:55:20,984] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step7000 is ready now!
0:   successfully saved checkpoint at iteration    7000 to checkpoints_280m5b9400m
7: time (ms) | save-checkpoint: 748.10
7:  iteration     7010/   11269 | consumed samples:      1794560 | consumed tokens:   3675258880 | elapsed time per iteration (s): 0.56 | learning rate: 7.733E-05 | global batch size:   256 | lm loss: 3.395437E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 456.048 | TFLOPs: 29.83 |
7:  iteration     7020/   11269 | consumed samples:      1797120 | consumed tokens:   3680501760 | elapsed time per iteration (s): 0.47 | learning rate: 7.710E-05 | global batch size:   256 | lm loss: 3.397600E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.887 | TFLOPs: 35.37 |
7:  iteration     7030/   11269 | consumed samples:      1799680 | consumed tokens:   3685744640 | elapsed time per iteration (s): 0.47 | learning rate: 7.686E-05 | global batch size:   256 | lm loss: 3.378922E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.932 | TFLOPs: 35.38 |
7:  iteration     7040/   11269 | consumed samples:      1802240 | consumed tokens:   3690987520 | elapsed time per iteration (s): 0.47 | learning rate: 7.662E-05 | global batch size:   256 | lm loss: 3.382222E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.487 | TFLOPs: 35.35 |
7:  iteration     7050/   11269 | consumed samples:      1804800 | consumed tokens:   3696230400 | elapsed time per iteration (s): 0.47 | learning rate: 7.639E-05 | global batch size:   256 | lm loss: 3.375137E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.575 | TFLOPs: 35.35 |
7:  iteration     7060/   11269 | consumed samples:      1807360 | consumed tokens:   3701473280 | elapsed time per iteration (s): 0.47 | learning rate: 7.615E-05 | global batch size:   256 | lm loss: 3.382994E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.589 | TFLOPs: 35.35 |
7:  iteration     7070/   11269 | consumed samples:      1809920 | consumed tokens:   3706716160 | elapsed time per iteration (s): 0.47 | learning rate: 7.592E-05 | global batch size:   256 | lm loss: 3.371759E+00 | grad norm: 0.326 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.376 | TFLOPs: 35.34 |
7:  iteration     7080/   11269 | consumed samples:      1812480 | consumed tokens:   3711959040 | elapsed time per iteration (s): 0.47 | learning rate: 7.569E-05 | global batch size:   256 | lm loss: 3.391137E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.332 | TFLOPs: 35.34 |
7:  iteration     7090/   11269 | consumed samples:      1815040 | consumed tokens:   3717201920 | elapsed time per iteration (s): 0.48 | learning rate: 7.545E-05 | global batch size:   256 | lm loss: 3.383102E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.329 | TFLOPs: 35.21 |
7:  iteration     7100/   11269 | consumed samples:      1817600 | consumed tokens:   3722444800 | elapsed time per iteration (s): 0.48 | learning rate: 7.522E-05 | global batch size:   256 | lm loss: 3.383538E+00 | grad norm: 0.286 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.830 | TFLOPs: 35.24 |
7:  iteration     7110/   11269 | consumed samples:      1820160 | consumed tokens:   3727687680 | elapsed time per iteration (s): 0.47 | learning rate: 7.498E-05 | global batch size:   256 | lm loss: 3.390726E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.853 | TFLOPs: 35.31 |
7:  iteration     7120/   11269 | consumed samples:      1822720 | consumed tokens:   3732930560 | elapsed time per iteration (s): 0.47 | learning rate: 7.475E-05 | global batch size:   256 | lm loss: 3.398857E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.867 | TFLOPs: 35.31 |
7:  iteration     7130/   11269 | consumed samples:      1825280 | consumed tokens:   3738173440 | elapsed time per iteration (s): 0.47 | learning rate: 7.452E-05 | global batch size:   256 | lm loss: 3.389166E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.822 | TFLOPs: 35.30 |
7:  iteration     7140/   11269 | consumed samples:      1827840 | consumed tokens:   3743416320 | elapsed time per iteration (s): 0.47 | learning rate: 7.428E-05 | global batch size:   256 | lm loss: 3.384270E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.866 | TFLOPs: 35.31 |
7:  iteration     7150/   11269 | consumed samples:      1830400 | consumed tokens:   3748659200 | elapsed time per iteration (s): 0.48 | learning rate: 7.405E-05 | global batch size:   256 | lm loss: 3.379140E+00 | grad norm: 0.363 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.531 | TFLOPs: 35.15 |
7:  iteration     7160/   11269 | consumed samples:      1832960 | consumed tokens:   3753902080 | elapsed time per iteration (s): 0.47 | learning rate: 7.382E-05 | global batch size:   256 | lm loss: 3.382446E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.670 | TFLOPs: 35.29 |
7:  iteration     7170/   11269 | consumed samples:      1835520 | consumed tokens:   3759144960 | elapsed time per iteration (s): 0.47 | learning rate: 7.359E-05 | global batch size:   256 | lm loss: 3.378856E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.927 | TFLOPs: 35.31 |
7:  iteration     7180/   11269 | consumed samples:      1838080 | consumed tokens:   3764387840 | elapsed time per iteration (s): 0.47 | learning rate: 7.336E-05 | global batch size:   256 | lm loss: 3.383644E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.311 | TFLOPs: 35.27 |
7:  iteration     7190/   11269 | consumed samples:      1840640 | consumed tokens:   3769630720 | elapsed time per iteration (s): 0.47 | learning rate: 7.313E-05 | global batch size:   256 | lm loss: 3.382055E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.589 | TFLOPs: 35.29 |
7:  iteration     7200/   11269 | consumed samples:      1843200 | consumed tokens:   3774873600 | elapsed time per iteration (s): 0.47 | learning rate: 7.289E-05 | global batch size:   256 | lm loss: 3.396053E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.540 | TFLOPs: 35.29 |
7:  iteration     7210/   11269 | consumed samples:      1845760 | consumed tokens:   3780116480 | elapsed time per iteration (s): 0.47 | learning rate: 7.266E-05 | global batch size:   256 | lm loss: 3.376611E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.856 | TFLOPs: 35.31 |
7:  iteration     7220/   11269 | consumed samples:      1848320 | consumed tokens:   3785359360 | elapsed time per iteration (s): 0.47 | learning rate: 7.243E-05 | global batch size:   256 | lm loss: 3.388765E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.709 | TFLOPs: 35.30 |
7:  iteration     7230/   11269 | consumed samples:      1850880 | consumed tokens:   3790602240 | elapsed time per iteration (s): 0.47 | learning rate: 7.220E-05 | global batch size:   256 | lm loss: 3.378211E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.569 | TFLOPs: 35.29 |
7:  iteration     7240/   11269 | consumed samples:      1853440 | consumed tokens:   3795845120 | elapsed time per iteration (s): 0.47 | learning rate: 7.197E-05 | global batch size:   256 | lm loss: 3.382515E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.998 | TFLOPs: 35.32 |
7:  iteration     7250/   11269 | consumed samples:      1856000 | consumed tokens:   3801088000 | elapsed time per iteration (s): 0.47 | learning rate: 7.174E-05 | global batch size:   256 | lm loss: 3.382980E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.509 | TFLOPs: 35.28 |
7:  iteration     7260/   11269 | consumed samples:      1858560 | consumed tokens:   3806330880 | elapsed time per iteration (s): 0.47 | learning rate: 7.151E-05 | global batch size:   256 | lm loss: 3.363499E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.576 | TFLOPs: 35.29 |
7:  iteration     7270/   11269 | consumed samples:      1861120 | consumed tokens:   3811573760 | elapsed time per iteration (s): 0.47 | learning rate: 7.129E-05 | global batch size:   256 | lm loss: 3.385050E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.542 | TFLOPs: 35.29 |
7:  iteration     7280/   11269 | consumed samples:      1863680 | consumed tokens:   3816816640 | elapsed time per iteration (s): 0.47 | learning rate: 7.106E-05 | global batch size:   256 | lm loss: 3.364845E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.694 | TFLOPs: 35.30 |
7:  iteration     7290/   11269 | consumed samples:      1866240 | consumed tokens:   3822059520 | elapsed time per iteration (s): 0.47 | learning rate: 7.083E-05 | global batch size:   256 | lm loss: 3.370649E+00 | grad norm: 0.285 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.545 | TFLOPs: 35.29 |
7:  iteration     7300/   11269 | consumed samples:      1868800 | consumed tokens:   3827302400 | elapsed time per iteration (s): 0.47 | learning rate: 7.060E-05 | global batch size:   256 | lm loss: 3.375064E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.609 | TFLOPs: 35.29 |
7:  iteration     7310/   11269 | consumed samples:      1871360 | consumed tokens:   3832545280 | elapsed time per iteration (s): 0.47 | learning rate: 7.037E-05 | global batch size:   256 | lm loss: 3.377546E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.747 | TFLOPs: 35.30 |
7:  iteration     7320/   11269 | consumed samples:      1873920 | consumed tokens:   3837788160 | elapsed time per iteration (s): 0.47 | learning rate: 7.014E-05 | global batch size:   256 | lm loss: 3.375941E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.750 | TFLOPs: 35.30 |
7:  iteration     7330/   11269 | consumed samples:      1876480 | consumed tokens:   3843031040 | elapsed time per iteration (s): 0.47 | learning rate: 6.992E-05 | global batch size:   256 | lm loss: 3.383419E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.553 | TFLOPs: 35.29 |
7:  iteration     7340/   11269 | consumed samples:      1879040 | consumed tokens:   3848273920 | elapsed time per iteration (s): 0.47 | learning rate: 6.969E-05 | global batch size:   256 | lm loss: 3.382812E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.871 | TFLOPs: 35.31 |
7:  iteration     7350/   11269 | consumed samples:      1881600 | consumed tokens:   3853516800 | elapsed time per iteration (s): 0.47 | learning rate: 6.946E-05 | global batch size:   256 | lm loss: 3.371232E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.608 | TFLOPs: 35.29 |
7:  iteration     7360/   11269 | consumed samples:      1884160 | consumed tokens:   3858759680 | elapsed time per iteration (s): 0.47 | learning rate: 6.924E-05 | global batch size:   256 | lm loss: 3.372501E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.036 | TFLOPs: 35.32 |
7:  iteration     7370/   11269 | consumed samples:      1886720 | consumed tokens:   3864002560 | elapsed time per iteration (s): 0.47 | learning rate: 6.901E-05 | global batch size:   256 | lm loss: 3.369599E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.509 | TFLOPs: 35.28 |
7:  iteration     7380/   11269 | consumed samples:      1889280 | consumed tokens:   3869245440 | elapsed time per iteration (s): 0.47 | learning rate: 6.879E-05 | global batch size:   256 | lm loss: 3.385218E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.247 | TFLOPs: 35.27 |
7:  iteration     7390/   11269 | consumed samples:      1891840 | consumed tokens:   3874488320 | elapsed time per iteration (s): 0.47 | learning rate: 6.856E-05 | global batch size:   256 | lm loss: 3.370727E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.894 | TFLOPs: 35.31 |
7:  iteration     7400/   11269 | consumed samples:      1894400 | consumed tokens:   3879731200 | elapsed time per iteration (s): 0.47 | learning rate: 6.834E-05 | global batch size:   256 | lm loss: 3.375203E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.571 | TFLOPs: 35.29 |
7:  iteration     7410/   11269 | consumed samples:      1896960 | consumed tokens:   3884974080 | elapsed time per iteration (s): 0.47 | learning rate: 6.811E-05 | global batch size:   256 | lm loss: 3.356388E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.512 | TFLOPs: 35.28 |
7:  iteration     7420/   11269 | consumed samples:      1899520 | consumed tokens:   3890216960 | elapsed time per iteration (s): 0.47 | learning rate: 6.789E-05 | global batch size:   256 | lm loss: 3.366221E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.725 | TFLOPs: 35.30 |
7:  iteration     7430/   11269 | consumed samples:      1902080 | consumed tokens:   3895459840 | elapsed time per iteration (s): 0.48 | learning rate: 6.766E-05 | global batch size:   256 | lm loss: 3.366190E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.560 | TFLOPs: 35.22 |
7:  iteration     7440/   11269 | consumed samples:      1904640 | consumed tokens:   3900702720 | elapsed time per iteration (s): 0.47 | learning rate: 6.744E-05 | global batch size:   256 | lm loss: 3.371812E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.815 | TFLOPs: 35.30 |
7:  iteration     7450/   11269 | consumed samples:      1907200 | consumed tokens:   3905945600 | elapsed time per iteration (s): 0.47 | learning rate: 6.722E-05 | global batch size:   256 | lm loss: 3.361149E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.599 | TFLOPs: 35.29 |
7:  iteration     7460/   11269 | consumed samples:      1909760 | consumed tokens:   3911188480 | elapsed time per iteration (s): 0.47 | learning rate: 6.700E-05 | global batch size:   256 | lm loss: 3.362653E+00 | grad norm: 0.285 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.432 | TFLOPs: 35.28 |
7:  iteration     7470/   11269 | consumed samples:      1912320 | consumed tokens:   3916431360 | elapsed time per iteration (s): 0.48 | learning rate: 6.677E-05 | global batch size:   256 | lm loss: 3.366566E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.427 | TFLOPs: 34.95 |
7:  iteration     7480/   11269 | consumed samples:      1914880 | consumed tokens:   3921674240 | elapsed time per iteration (s): 0.48 | learning rate: 6.655E-05 | global batch size:   256 | lm loss: 3.370388E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.242 | TFLOPs: 34.74 |
7:  iteration     7490/   11269 | consumed samples:      1917440 | consumed tokens:   3926917120 | elapsed time per iteration (s): 0.48 | learning rate: 6.633E-05 | global batch size:   256 | lm loss: 3.372507E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.201 | TFLOPs: 35.20 |
7:  iteration     7500/   11269 | consumed samples:      1920000 | consumed tokens:   3932160000 | elapsed time per iteration (s): 0.48 | learning rate: 6.611E-05 | global batch size:   256 | lm loss: 3.371844E+00 | grad norm: 0.286 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.788 | TFLOPs: 35.24 |
7:  iteration     7510/   11269 | consumed samples:      1922560 | consumed tokens:   3937402880 | elapsed time per iteration (s): 0.47 | learning rate: 6.589E-05 | global batch size:   256 | lm loss: 3.371706E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.136 | TFLOPs: 35.32 |
7:  iteration     7520/   11269 | consumed samples:      1925120 | consumed tokens:   3942645760 | elapsed time per iteration (s): 0.47 | learning rate: 6.567E-05 | global batch size:   256 | lm loss: 3.365607E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.703 | TFLOPs: 35.30 |
7:  iteration     7530/   11269 | consumed samples:      1927680 | consumed tokens:   3947888640 | elapsed time per iteration (s): 0.47 | learning rate: 6.545E-05 | global batch size:   256 | lm loss: 3.370324E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.211 | TFLOPs: 35.33 |
7:  iteration     7540/   11269 | consumed samples:      1930240 | consumed tokens:   3953131520 | elapsed time per iteration (s): 0.47 | learning rate: 6.523E-05 | global batch size:   256 | lm loss: 3.383733E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.663 | TFLOPs: 35.29 |
7:  iteration     7550/   11269 | consumed samples:      1932800 | consumed tokens:   3958374400 | elapsed time per iteration (s): 0.47 | learning rate: 6.501E-05 | global batch size:   256 | lm loss: 3.382576E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.715 | TFLOPs: 35.30 |
7:  iteration     7560/   11269 | consumed samples:      1935360 | consumed tokens:   3963617280 | elapsed time per iteration (s): 0.47 | learning rate: 6.479E-05 | global batch size:   256 | lm loss: 3.381438E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.046 | TFLOPs: 35.32 |
7:  iteration     7570/   11269 | consumed samples:      1937920 | consumed tokens:   3968860160 | elapsed time per iteration (s): 0.47 | learning rate: 6.457E-05 | global batch size:   256 | lm loss: 3.369043E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.602 | TFLOPs: 35.29 |
7:  iteration     7580/   11269 | consumed samples:      1940480 | consumed tokens:   3974103040 | elapsed time per iteration (s): 0.47 | learning rate: 6.435E-05 | global batch size:   256 | lm loss: 3.374106E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.564 | TFLOPs: 35.29 |
7:  iteration     7590/   11269 | consumed samples:      1943040 | consumed tokens:   3979345920 | elapsed time per iteration (s): 0.47 | learning rate: 6.413E-05 | global batch size:   256 | lm loss: 3.369818E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.163 | TFLOPs: 35.26 |
7:  iteration     7600/   11269 | consumed samples:      1945600 | consumed tokens:   3984588800 | elapsed time per iteration (s): 0.47 | learning rate: 6.391E-05 | global batch size:   256 | lm loss: 3.359015E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.533 | TFLOPs: 35.29 |
7:  iteration     7610/   11269 | consumed samples:      1948160 | consumed tokens:   3989831680 | elapsed time per iteration (s): 0.47 | learning rate: 6.370E-05 | global batch size:   256 | lm loss: 3.376344E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.005 | TFLOPs: 35.25 |
7:  iteration     7620/   11269 | consumed samples:      1950720 | consumed tokens:   3995074560 | elapsed time per iteration (s): 0.47 | learning rate: 6.348E-05 | global batch size:   256 | lm loss: 3.368158E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.877 | TFLOPs: 35.31 |
7:  iteration     7630/   11269 | consumed samples:      1953280 | consumed tokens:   4000317440 | elapsed time per iteration (s): 0.47 | learning rate: 6.326E-05 | global batch size:   256 | lm loss: 3.371413E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.768 | TFLOPs: 35.30 |
7:  iteration     7640/   11269 | consumed samples:      1955840 | consumed tokens:   4005560320 | elapsed time per iteration (s): 0.47 | learning rate: 6.305E-05 | global batch size:   256 | lm loss: 3.366294E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.776 | TFLOPs: 35.30 |
7:  iteration     7650/   11269 | consumed samples:      1958400 | consumed tokens:   4010803200 | elapsed time per iteration (s): 0.47 | learning rate: 6.283E-05 | global batch size:   256 | lm loss: 3.361715E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.383 | TFLOPs: 35.28 |
7:  iteration     7660/   11269 | consumed samples:      1960960 | consumed tokens:   4016046080 | elapsed time per iteration (s): 0.47 | learning rate: 6.261E-05 | global batch size:   256 | lm loss: 3.358666E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.634 | TFLOPs: 35.29 |
7:  iteration     7670/   11269 | consumed samples:      1963520 | consumed tokens:   4021288960 | elapsed time per iteration (s): 0.47 | learning rate: 6.240E-05 | global batch size:   256 | lm loss: 3.365503E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.611 | TFLOPs: 35.29 |
7:  iteration     7680/   11269 | consumed samples:      1966080 | consumed tokens:   4026531840 | elapsed time per iteration (s): 0.47 | learning rate: 6.218E-05 | global batch size:   256 | lm loss: 3.370166E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.396 | TFLOPs: 35.28 |
7:  iteration     7690/   11269 | consumed samples:      1968640 | consumed tokens:   4031774720 | elapsed time per iteration (s): 0.47 | learning rate: 6.197E-05 | global batch size:   256 | lm loss: 3.363000E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.656 | TFLOPs: 35.29 |
7:  iteration     7700/   11269 | consumed samples:      1971200 | consumed tokens:   4037017600 | elapsed time per iteration (s): 0.47 | learning rate: 6.175E-05 | global batch size:   256 | lm loss: 3.351837E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.519 | TFLOPs: 35.28 |
7:  iteration     7710/   11269 | consumed samples:      1973760 | consumed tokens:   4042260480 | elapsed time per iteration (s): 0.47 | learning rate: 6.154E-05 | global batch size:   256 | lm loss: 3.362171E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.504 | TFLOPs: 35.28 |
7:  iteration     7720/   11269 | consumed samples:      1976320 | consumed tokens:   4047503360 | elapsed time per iteration (s): 0.47 | learning rate: 6.133E-05 | global batch size:   256 | lm loss: 3.368191E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.797 | TFLOPs: 35.30 |
7:  iteration     7730/   11269 | consumed samples:      1978880 | consumed tokens:   4052746240 | elapsed time per iteration (s): 0.47 | learning rate: 6.111E-05 | global batch size:   256 | lm loss: 3.369874E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.484 | TFLOPs: 35.28 |
7:  iteration     7740/   11269 | consumed samples:      1981440 | consumed tokens:   4057989120 | elapsed time per iteration (s): 0.47 | learning rate: 6.090E-05 | global batch size:   256 | lm loss: 3.353785E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.710 | TFLOPs: 35.30 |
7:  iteration     7750/   11269 | consumed samples:      1984000 | consumed tokens:   4063232000 | elapsed time per iteration (s): 0.47 | learning rate: 6.069E-05 | global batch size:   256 | lm loss: 3.379349E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.633 | TFLOPs: 35.29 |
7:  iteration     7760/   11269 | consumed samples:      1986560 | consumed tokens:   4068474880 | elapsed time per iteration (s): 0.47 | learning rate: 6.048E-05 | global batch size:   256 | lm loss: 3.376830E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.489 | TFLOPs: 35.28 |
7:  iteration     7770/   11269 | consumed samples:      1989120 | consumed tokens:   4073717760 | elapsed time per iteration (s): 0.47 | learning rate: 6.027E-05 | global batch size:   256 | lm loss: 3.373725E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.863 | TFLOPs: 35.31 |
7:  iteration     7780/   11269 | consumed samples:      1991680 | consumed tokens:   4078960640 | elapsed time per iteration (s): 0.47 | learning rate: 6.006E-05 | global batch size:   256 | lm loss: 3.362823E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.562 | TFLOPs: 35.29 |
7:  iteration     7790/   11269 | consumed samples:      1994240 | consumed tokens:   4084203520 | elapsed time per iteration (s): 0.47 | learning rate: 5.984E-05 | global batch size:   256 | lm loss: 3.360585E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.861 | TFLOPs: 35.31 |
7:  iteration     7800/   11269 | consumed samples:      1996800 | consumed tokens:   4089446400 | elapsed time per iteration (s): 0.47 | learning rate: 5.963E-05 | global batch size:   256 | lm loss: 3.362442E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.146 | TFLOPs: 35.33 |
7:  iteration     7810/   11269 | consumed samples:      1999360 | consumed tokens:   4094689280 | elapsed time per iteration (s): 0.47 | learning rate: 5.942E-05 | global batch size:   256 | lm loss: 3.366068E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.005 | TFLOPs: 35.32 |
7:  iteration     7820/   11269 | consumed samples:      2001920 | consumed tokens:   4099932160 | elapsed time per iteration (s): 0.47 | learning rate: 5.922E-05 | global batch size:   256 | lm loss: 3.360478E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.545 | TFLOPs: 35.29 |
7:  iteration     7830/   11269 | consumed samples:      2004480 | consumed tokens:   4105175040 | elapsed time per iteration (s): 0.47 | learning rate: 5.901E-05 | global batch size:   256 | lm loss: 3.359552E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.071 | TFLOPs: 35.32 |
7:  iteration     7840/   11269 | consumed samples:      2007040 | consumed tokens:   4110417920 | elapsed time per iteration (s): 0.47 | learning rate: 5.880E-05 | global batch size:   256 | lm loss: 3.357920E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.870 | TFLOPs: 35.31 |
7:  iteration     7850/   11269 | consumed samples:      2009600 | consumed tokens:   4115660800 | elapsed time per iteration (s): 0.47 | learning rate: 5.859E-05 | global batch size:   256 | lm loss: 3.355674E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.998 | TFLOPs: 35.32 |
7:  iteration     7860/   11269 | consumed samples:      2012160 | consumed tokens:   4120903680 | elapsed time per iteration (s): 0.47 | learning rate: 5.838E-05 | global batch size:   256 | lm loss: 3.355836E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.612 | TFLOPs: 35.29 |
7:  iteration     7870/   11269 | consumed samples:      2014720 | consumed tokens:   4126146560 | elapsed time per iteration (s): 0.47 | learning rate: 5.817E-05 | global batch size:   256 | lm loss: 3.367831E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.695 | TFLOPs: 35.30 |
7:  iteration     7880/   11269 | consumed samples:      2017280 | consumed tokens:   4131389440 | elapsed time per iteration (s): 0.47 | learning rate: 5.797E-05 | global batch size:   256 | lm loss: 3.365918E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.765 | TFLOPs: 35.30 |
7:  iteration     7890/   11269 | consumed samples:      2019840 | consumed tokens:   4136632320 | elapsed time per iteration (s): 0.47 | learning rate: 5.776E-05 | global batch size:   256 | lm loss: 3.358317E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.041 | TFLOPs: 35.32 |
7:  iteration     7900/   11269 | consumed samples:      2022400 | consumed tokens:   4141875200 | elapsed time per iteration (s): 0.47 | learning rate: 5.755E-05 | global batch size:   256 | lm loss: 3.363395E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.590 | TFLOPs: 35.29 |
7:  iteration     7910/   11269 | consumed samples:      2024960 | consumed tokens:   4147118080 | elapsed time per iteration (s): 0.47 | learning rate: 5.735E-05 | global batch size:   256 | lm loss: 3.362957E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.897 | TFLOPs: 35.31 |
7:  iteration     7920/   11269 | consumed samples:      2027520 | consumed tokens:   4152360960 | elapsed time per iteration (s): 0.47 | learning rate: 5.714E-05 | global batch size:   256 | lm loss: 3.350002E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.743 | TFLOPs: 35.30 |
7:  iteration     7930/   11269 | consumed samples:      2030080 | consumed tokens:   4157603840 | elapsed time per iteration (s): 0.47 | learning rate: 5.694E-05 | global batch size:   256 | lm loss: 3.352530E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.667 | TFLOPs: 35.29 |
7:  iteration     7940/   11269 | consumed samples:      2032640 | consumed tokens:   4162846720 | elapsed time per iteration (s): 0.47 | learning rate: 5.673E-05 | global batch size:   256 | lm loss: 3.355146E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.901 | TFLOPs: 35.31 |
7:  iteration     7950/   11269 | consumed samples:      2035200 | consumed tokens:   4168089600 | elapsed time per iteration (s): 0.47 | learning rate: 5.653E-05 | global batch size:   256 | lm loss: 3.352403E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.826 | TFLOPs: 35.30 |
7:  iteration     7960/   11269 | consumed samples:      2037760 | consumed tokens:   4173332480 | elapsed time per iteration (s): 0.47 | learning rate: 5.633E-05 | global batch size:   256 | lm loss: 3.349899E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.618 | TFLOPs: 35.29 |
7:  iteration     7970/   11269 | consumed samples:      2040320 | consumed tokens:   4178575360 | elapsed time per iteration (s): 0.49 | learning rate: 5.612E-05 | global batch size:   256 | lm loss: 3.344794E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 524.333 | TFLOPs: 34.29 |
7:  iteration     7980/   11269 | consumed samples:      2042880 | consumed tokens:   4183818240 | elapsed time per iteration (s): 0.49 | learning rate: 5.592E-05 | global batch size:   256 | lm loss: 3.356477E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 520.993 | TFLOPs: 34.07 |
7:  iteration     7990/   11269 | consumed samples:      2045440 | consumed tokens:   4189061120 | elapsed time per iteration (s): 0.47 | learning rate: 5.572E-05 | global batch size:   256 | lm loss: 3.348618E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.207 | TFLOPs: 35.26 |
0: [2023-03-15 23:03:15,793] [INFO] [logging.py:68:log_dist] [Rank 0] step=8000, skipped=0, lr=[5.551588445114958e-05, 5.551588445114958e-05, 5.551588445114958e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration     8000/   11269 | consumed samples:      2048000 | consumed tokens:   4194304000 | elapsed time per iteration (s): 0.48 | learning rate: 5.552E-05 | global batch size:   256 | lm loss: 3.341730E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.113 | TFLOPs: 34.93 |
0: steps: 8000 loss: 3.3186 iter time (s): 0.473 samples/sec: 541.741
7: -----------------------------------------------------------------------------------------------
7: validation loss at iteration 8000 | lm loss value: 3.467995E+00 | lm loss PPL: 3.207236E+01 | 
7: -----------------------------------------------------------------------------------------------
0: saving checkpoint at iteration    8000 to checkpoints_280m5b9400m
0: [2023-03-15 23:03:15,973] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step8000 is begin to save!
0: [2023-03-15 23:03:15,977] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_01-model_00-model_states.pt...
0: [2023-03-15 23:03:16,106] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_01-model_00-model_states.pt.
0: [2023-03-15 23:03:16,106] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_03-model_00-model_states.pt...
0: [2023-03-15 23:03:16,135] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_03-model_00-model_states.pt.
0: [2023-03-15 23:03:16,135] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_04-model_00-model_states.pt...
0: [2023-03-15 23:03:16,161] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_04-model_00-model_states.pt.
0: [2023-03-15 23:03:16,161] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_05-model_00-model_states.pt...
0: [2023-03-15 23:03:16,188] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_05-model_00-model_states.pt.
0: [2023-03-15 23:03:16,188] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_06-model_00-model_states.pt...
0: [2023-03-15 23:03:16,214] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_06-model_00-model_states.pt.
0: [2023-03-15 23:03:16,215] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_07-model_00-model_states.pt...
0: [2023-03-15 23:03:16,239] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_07-model_00-model_states.pt.
0: [2023-03-15 23:03:16,239] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_08-model_00-model_states.pt...
0: [2023-03-15 23:03:16,265] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_08-model_00-model_states.pt.
0: [2023-03-15 23:03:16,265] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_09-model_00-model_states.pt...
0: [2023-03-15 23:03:16,292] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_09-model_00-model_states.pt.
0: [2023-03-15 23:03:16,292] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_10-model_00-model_states.pt...
0: [2023-03-15 23:03:16,320] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_10-model_00-model_states.pt.
0: [2023-03-15 23:03:16,320] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_11-model_00-model_states.pt...
0: [2023-03-15 23:03:16,346] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_11-model_00-model_states.pt.
0: [2023-03-15 23:03:16,346] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_12-model_00-model_states.pt...
0: [2023-03-15 23:03:16,372] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_12-model_00-model_states.pt.
0: [2023-03-15 23:03:16,372] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_13-model_00-model_states.pt...
0: [2023-03-15 23:03:16,397] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_13-model_00-model_states.pt.
0: [2023-03-15 23:03:16,398] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_14-model_00-model_states.pt...
0: [2023-03-15 23:03:16,424] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_14-model_00-model_states.pt.
0: [2023-03-15 23:03:16,424] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_15-model_00-model_states.pt...
0: [2023-03-15 23:03:16,450] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_15-model_00-model_states.pt.
0: [2023-03-15 23:03:16,450] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_16-model_00-model_states.pt...
0: [2023-03-15 23:03:16,476] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_16-model_00-model_states.pt.
0: [2023-03-15 23:03:16,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_17-model_00-model_states.pt...
0: [2023-03-15 23:03:16,502] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_17-model_00-model_states.pt.
0: [2023-03-15 23:03:16,502] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_18-model_00-model_states.pt...
0: [2023-03-15 23:03:16,528] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_18-model_00-model_states.pt.
0: [2023-03-15 23:03:16,528] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_19-model_00-model_states.pt...
0: [2023-03-15 23:03:16,555] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_19-model_00-model_states.pt.
0: [2023-03-15 23:03:16,556] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_20-model_00-model_states.pt...
0: [2023-03-15 23:03:16,582] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_20-model_00-model_states.pt.
0: [2023-03-15 23:03:16,582] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/layer_22-model_00-model_states.pt...
0: [2023-03-15 23:03:16,586] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/layer_22-model_00-model_states.pt.
0: [2023-03-15 23:03:16,587] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_280m5b9400m/global_step8000/mp_rank_00_model_states.pt
0: [2023-03-15 23:03:16,587] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/mp_rank_00_model_states.pt...
0: [2023-03-15 23:03:16,590] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/mp_rank_00_model_states.pt.
0: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:03:16,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:03:16,684] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:03:16,689] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:03:16,689] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-15 23:03:16,689] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
6: [2023-03-15 23:03:16,691] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:03:16,691] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-15 23:03:16,691] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
0: [2023-03-15 23:03:16,691] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:03:16,691] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-15 23:03:16,692] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
0: [2023-03-15 23:03:16,691] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:03:16,692] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-15 23:03:16,692] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
3: [2023-03-15 23:03:16,693] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:03:16,696] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:03:16,696] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-15 23:03:16,696] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
0: [2023-03-15 23:03:16,696] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:03:16,696] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-15 23:03:16,696] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
6: [2023-03-15 23:03:16,698] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:03:16,698] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-15 23:03:16,698] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
4: [2023-03-15 23:03:16,699] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:03:16,699] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-15 23:03:16,699] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
4: [2023-03-15 23:03:16,699] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:03:16,699] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-15 23:03:16,700] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
4: [2023-03-15 23:03:16,700] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:03:16,700] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-15 23:03:16,700] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
6: [2023-03-15 23:03:16,700] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:03:16,700] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:03:16,700] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-15 23:03:16,700] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-15 23:03:16,700] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
6: [2023-03-15 23:03:16,700] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
4: [2023-03-15 23:03:16,700] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:03:16,700] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-15 23:03:16,701] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
7: [2023-03-15 23:03:16,702] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:03:16,702] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:03:16,702] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-15 23:03:16,702] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-15 23:03:16,702] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
7: [2023-03-15 23:03:16,702] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
7: [2023-03-15 23:03:16,702] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:03:16,702] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:03:16,702] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:03:16,702] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-15 23:03:16,702] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-15 23:03:16,702] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-15 23:03:16,702] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
7: [2023-03-15 23:03:16,702] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
7: [2023-03-15 23:03:16,702] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
7: [2023-03-15 23:03:16,704] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:03:16,704] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-15 23:03:16,704] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
6: [2023-03-15 23:03:16,705] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:03:16,705] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-15 23:03:16,705] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
7: [2023-03-15 23:03:16,708] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:03:16,708] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-15 23:03:16,708] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
6: [2023-03-15 23:03:16,708] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:03:16,709] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-15 23:03:16,709] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
6: [2023-03-15 23:03:16,711] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:03:16,711] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-15 23:03:16,711] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
4: [2023-03-15 23:03:16,704] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:03:16,705] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-15 23:03:16,705] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
2: [2023-03-15 23:03:16,712] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:03:16,712] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:03:16,712] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:03:16,712] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-15 23:03:16,712] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
2: [2023-03-15 23:03:16,712] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-15 23:03:16,712] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-15 23:03:16,712] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
2: [2023-03-15 23:03:16,712] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
5: [2023-03-15 23:03:16,714] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:03:16,716] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:03:16,716] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-15 23:03:16,716] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
4: [2023-03-15 23:03:16,715] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:03:16,715] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-15 23:03:16,715] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
4: [2023-03-15 23:03:16,715] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:03:16,715] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-15 23:03:16,715] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
4: [2023-03-15 23:03:16,718] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:03:16,718] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-15 23:03:16,718] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
0: [2023-03-15 23:03:16,718] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:03:16,718] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-15 23:03:16,718] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
0: [2023-03-15 23:03:16,719] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:03:16,719] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-15 23:03:16,719] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
0: [2023-03-15 23:03:16,720] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:03:16,720] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-15 23:03:16,720] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
2: [2023-03-15 23:03:16,722] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:03:16,723] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-15 23:03:16,723] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
2: [2023-03-15 23:03:16,724] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:03:16,724] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:03:16,724] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:03:16,724] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:03:16,724] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-15 23:03:16,724] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-15 23:03:16,724] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-15 23:03:16,724] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-15 23:03:16,724] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
2: [2023-03-15 23:03:16,724] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
2: [2023-03-15 23:03:16,724] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
2: [2023-03-15 23:03:16,724] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
6: [2023-03-15 23:03:16,731] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:03:16,731] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-15 23:03:16,731] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
1: [2023-03-15 23:03:16,732] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:03:16,732] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:03:16,732] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-15 23:03:16,732] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-15 23:03:16,732] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
1: [2023-03-15 23:03:16,732] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
1: [2023-03-15 23:03:16,734] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:03:16,734] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:03:16,734] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:03:16,734] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:03:16,734] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-15 23:03:16,734] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
1: [2023-03-15 23:03:16,734] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-15 23:03:16,734] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-15 23:03:16,734] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-15 23:03:16,734] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
1: [2023-03-15 23:03:16,734] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
1: [2023-03-15 23:03:16,734] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
3: [2023-03-15 23:03:16,693] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-15 23:03:16,693] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
3: [2023-03-15 23:03:16,697] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:03:16,697] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-15 23:03:16,697] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
3: [2023-03-15 23:03:16,702] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:03:16,702] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:03:16,702] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-15 23:03:16,702] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-15 23:03:16,702] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
3: [2023-03-15 23:03:16,702] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
3: [2023-03-15 23:03:16,708] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:03:16,708] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-15 23:03:16,708] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
3: [2023-03-15 23:03:16,708] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:03:16,708] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-15 23:03:16,708] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
3: [2023-03-15 23:03:16,713] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:03:16,714] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-15 23:03:16,714] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
3: [2023-03-15 23:03:16,714] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:03:16,714] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-15 23:03:16,714] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
5: [2023-03-15 23:03:16,714] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-15 23:03:16,714] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
5: [2023-03-15 23:03:16,715] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:03:16,715] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-15 23:03:16,715] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
5: [2023-03-15 23:03:16,715] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:03:16,715] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-15 23:03:16,715] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
5: [2023-03-15 23:03:16,716] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:03:16,716] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-15 23:03:16,716] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
5: [2023-03-15 23:03:16,716] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:03:16,716] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-15 23:03:16,717] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
5: [2023-03-15 23:03:16,724] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:03:16,724] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-15 23:03:16,724] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
5: [2023-03-15 23:03:16,725] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:03:16,725] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-15 23:03:16,725] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
5: [2023-03-15 23:03:16,725] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:03:16,725] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-15 23:03:16,725] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
0: [2023-03-15 23:03:16,762] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:03:16,762] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-15 23:03:16,762] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
0: [2023-03-15 23:03:16,783] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step8000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-15 23:03:16,783] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step8000 is ready now!
0:   successfully saved checkpoint at iteration    8000 to checkpoints_280m5b9400m
7: time (ms) | save-checkpoint: 817.74
7:  iteration     8010/   11269 | consumed samples:      2050560 | consumed tokens:   4199546880 | elapsed time per iteration (s): 0.59 | learning rate: 5.531E-05 | global batch size:   256 | lm loss: 3.346957E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 432.532 | TFLOPs: 28.29 |
7:  iteration     8020/   11269 | consumed samples:      2053120 | consumed tokens:   4204789760 | elapsed time per iteration (s): 0.48 | learning rate: 5.511E-05 | global batch size:   256 | lm loss: 3.352927E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 532.893 | TFLOPs: 34.85 |
7:  iteration     8030/   11269 | consumed samples:      2055680 | consumed tokens:   4210032640 | elapsed time per iteration (s): 0.49 | learning rate: 5.491E-05 | global batch size:   256 | lm loss: 3.351870E+00 | grad norm: 0.284 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 524.506 | TFLOPs: 34.30 |
7:  iteration     8040/   11269 | consumed samples:      2058240 | consumed tokens:   4215275520 | elapsed time per iteration (s): 0.47 | learning rate: 5.471E-05 | global batch size:   256 | lm loss: 3.357246E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.291 | TFLOPs: 35.40 |
7:  iteration     8050/   11269 | consumed samples:      2060800 | consumed tokens:   4220518400 | elapsed time per iteration (s): 0.47 | learning rate: 5.451E-05 | global batch size:   256 | lm loss: 3.346156E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.136 | TFLOPs: 35.39 |
7:  iteration     8060/   11269 | consumed samples:      2063360 | consumed tokens:   4225761280 | elapsed time per iteration (s): 0.47 | learning rate: 5.431E-05 | global batch size:   256 | lm loss: 3.348359E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.963 | TFLOPs: 35.38 |
7:  iteration     8070/   11269 | consumed samples:      2065920 | consumed tokens:   4231004160 | elapsed time per iteration (s): 0.47 | learning rate: 5.411E-05 | global batch size:   256 | lm loss: 3.352924E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.081 | TFLOPs: 35.39 |
7:  iteration     8080/   11269 | consumed samples:      2068480 | consumed tokens:   4236247040 | elapsed time per iteration (s): 0.47 | learning rate: 5.392E-05 | global batch size:   256 | lm loss: 3.359683E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.536 | TFLOPs: 35.35 |
7:  iteration     8090/   11269 | consumed samples:      2071040 | consumed tokens:   4241489920 | elapsed time per iteration (s): 0.51 | learning rate: 5.372E-05 | global batch size:   256 | lm loss: 3.353542E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 497.669 | TFLOPs: 32.55 |
7:  iteration     8100/   11269 | consumed samples:      2073600 | consumed tokens:   4246732800 | elapsed time per iteration (s): 0.55 | learning rate: 5.352E-05 | global batch size:   256 | lm loss: 3.357018E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 462.617 | TFLOPs: 30.26 |
7:  iteration     8110/   11269 | consumed samples:      2076160 | consumed tokens:   4251975680 | elapsed time per iteration (s): 0.57 | learning rate: 5.332E-05 | global batch size:   256 | lm loss: 3.341851E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 449.346 | TFLOPs: 29.39 |
7:  iteration     8120/   11269 | consumed samples:      2078720 | consumed tokens:   4257218560 | elapsed time per iteration (s): 0.51 | learning rate: 5.313E-05 | global batch size:   256 | lm loss: 3.350813E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 501.924 | TFLOPs: 32.83 |
7:  iteration     8130/   11269 | consumed samples:      2081280 | consumed tokens:   4262461440 | elapsed time per iteration (s): 0.48 | learning rate: 5.293E-05 | global batch size:   256 | lm loss: 3.342475E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.689 | TFLOPs: 34.90 |
7:  iteration     8140/   11269 | consumed samples:      2083840 | consumed tokens:   4267704320 | elapsed time per iteration (s): 0.48 | learning rate: 5.273E-05 | global batch size:   256 | lm loss: 3.350589E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 530.813 | TFLOPs: 34.71 |
7:  iteration     8150/   11269 | consumed samples:      2086400 | consumed tokens:   4272947200 | elapsed time per iteration (s): 0.48 | learning rate: 5.254E-05 | global batch size:   256 | lm loss: 3.352605E+00 | grad norm: 0.326 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.590 | TFLOPs: 34.77 |
7:  iteration     8160/   11269 | consumed samples:      2088960 | consumed tokens:   4278190080 | elapsed time per iteration (s): 0.49 | learning rate: 5.234E-05 | global batch size:   256 | lm loss: 3.348596E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 522.621 | TFLOPs: 34.18 |
7:  iteration     8170/   11269 | consumed samples:      2091520 | consumed tokens:   4283432960 | elapsed time per iteration (s): 0.51 | learning rate: 5.215E-05 | global batch size:   256 | lm loss: 3.348682E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 499.077 | TFLOPs: 32.64 |
7:  iteration     8180/   11269 | consumed samples:      2094080 | consumed tokens:   4288675840 | elapsed time per iteration (s): 0.48 | learning rate: 5.196E-05 | global batch size:   256 | lm loss: 3.347631E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.698 | TFLOPs: 35.03 |
7:  iteration     8190/   11269 | consumed samples:      2096640 | consumed tokens:   4293918720 | elapsed time per iteration (s): 0.48 | learning rate: 5.176E-05 | global batch size:   256 | lm loss: 3.339325E+00 | grad norm: 0.330 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.827 | TFLOPs: 35.17 |
7:  iteration     8200/   11269 | consumed samples:      2099200 | consumed tokens:   4299161600 | elapsed time per iteration (s): 0.48 | learning rate: 5.157E-05 | global batch size:   256 | lm loss: 3.347945E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.128 | TFLOPs: 35.00 |
7:  iteration     8210/   11269 | consumed samples:      2101760 | consumed tokens:   4304404480 | elapsed time per iteration (s): 0.48 | learning rate: 5.138E-05 | global batch size:   256 | lm loss: 3.348700E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.891 | TFLOPs: 35.11 |
7:  iteration     8220/   11269 | consumed samples:      2104320 | consumed tokens:   4309647360 | elapsed time per iteration (s): 0.47 | learning rate: 5.119E-05 | global batch size:   256 | lm loss: 3.350323E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.608 | TFLOPs: 35.29 |
7:  iteration     8230/   11269 | consumed samples:      2106880 | consumed tokens:   4314890240 | elapsed time per iteration (s): 0.47 | learning rate: 5.099E-05 | global batch size:   256 | lm loss: 3.351265E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.976 | TFLOPs: 35.31 |
7:  iteration     8240/   11269 | consumed samples:      2109440 | consumed tokens:   4320133120 | elapsed time per iteration (s): 0.47 | learning rate: 5.080E-05 | global batch size:   256 | lm loss: 3.337860E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.009 | TFLOPs: 35.38 |
7:  iteration     8250/   11269 | consumed samples:      2112000 | consumed tokens:   4325376000 | elapsed time per iteration (s): 0.48 | learning rate: 5.061E-05 | global batch size:   256 | lm loss: 3.333883E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.154 | TFLOPs: 35.06 |
7:  iteration     8260/   11269 | consumed samples:      2114560 | consumed tokens:   4330618880 | elapsed time per iteration (s): 0.48 | learning rate: 5.042E-05 | global batch size:   256 | lm loss: 3.340737E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 532.900 | TFLOPs: 34.85 |
7:  iteration     8270/   11269 | consumed samples:      2117120 | consumed tokens:   4335861760 | elapsed time per iteration (s): 0.48 | learning rate: 5.023E-05 | global batch size:   256 | lm loss: 3.340205E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.973 | TFLOPs: 34.99 |
7:  iteration     8280/   11269 | consumed samples:      2119680 | consumed tokens:   4341104640 | elapsed time per iteration (s): 0.48 | learning rate: 5.004E-05 | global batch size:   256 | lm loss: 3.346501E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.929 | TFLOPs: 35.05 |
7:  iteration     8290/   11269 | consumed samples:      2122240 | consumed tokens:   4346347520 | elapsed time per iteration (s): 0.49 | learning rate: 4.985E-05 | global batch size:   256 | lm loss: 3.326854E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 519.795 | TFLOPs: 33.99 |
7:  iteration     8300/   11269 | consumed samples:      2124800 | consumed tokens:   4351590400 | elapsed time per iteration (s): 0.49 | learning rate: 4.967E-05 | global batch size:   256 | lm loss: 3.335303E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 526.075 | TFLOPs: 34.41 |
7:  iteration     8310/   11269 | consumed samples:      2127360 | consumed tokens:   4356833280 | elapsed time per iteration (s): 0.48 | learning rate: 4.948E-05 | global batch size:   256 | lm loss: 3.333305E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.472 | TFLOPs: 35.09 |
7:  iteration     8320/   11269 | consumed samples:      2129920 | consumed tokens:   4362076160 | elapsed time per iteration (s): 0.48 | learning rate: 4.929E-05 | global batch size:   256 | lm loss: 3.343834E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 530.003 | TFLOPs: 34.66 |
7:  iteration     8330/   11269 | consumed samples:      2132480 | consumed tokens:   4367319040 | elapsed time per iteration (s): 0.49 | learning rate: 4.910E-05 | global batch size:   256 | lm loss: 3.353397E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 526.724 | TFLOPs: 34.45 |
7:  iteration     8340/   11269 | consumed samples:      2135040 | consumed tokens:   4372561920 | elapsed time per iteration (s): 0.48 | learning rate: 4.892E-05 | global batch size:   256 | lm loss: 3.334957E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.632 | TFLOPs: 34.77 |
7:  iteration     8350/   11269 | consumed samples:      2137600 | consumed tokens:   4377804800 | elapsed time per iteration (s): 0.48 | learning rate: 4.873E-05 | global batch size:   256 | lm loss: 3.348095E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 528.331 | TFLOPs: 34.55 |
7:  iteration     8360/   11269 | consumed samples:      2140160 | consumed tokens:   4383047680 | elapsed time per iteration (s): 0.48 | learning rate: 4.855E-05 | global batch size:   256 | lm loss: 3.347230E+00 | grad norm: 0.284 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 528.318 | TFLOPs: 34.55 |
7:  iteration     8370/   11269 | consumed samples:      2142720 | consumed tokens:   4388290560 | elapsed time per iteration (s): 0.47 | learning rate: 4.836E-05 | global batch size:   256 | lm loss: 3.333686E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.319 | TFLOPs: 35.40 |
7:  iteration     8380/   11269 | consumed samples:      2145280 | consumed tokens:   4393533440 | elapsed time per iteration (s): 0.48 | learning rate: 4.818E-05 | global batch size:   256 | lm loss: 3.337275E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.212 | TFLOPs: 34.74 |
7:  iteration     8390/   11269 | consumed samples:      2147840 | consumed tokens:   4398776320 | elapsed time per iteration (s): 0.49 | learning rate: 4.799E-05 | global batch size:   256 | lm loss: 3.340240E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 520.735 | TFLOPs: 34.06 |
7:  iteration     8400/   11269 | consumed samples:      2150400 | consumed tokens:   4404019200 | elapsed time per iteration (s): 0.50 | learning rate: 4.781E-05 | global batch size:   256 | lm loss: 3.331691E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 509.269 | TFLOPs: 33.31 |
7:  iteration     8410/   11269 | consumed samples:      2152960 | consumed tokens:   4409262080 | elapsed time per iteration (s): 0.47 | learning rate: 4.763E-05 | global batch size:   256 | lm loss: 3.345345E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.091 | TFLOPs: 35.32 |
7:  iteration     8420/   11269 | consumed samples:      2155520 | consumed tokens:   4414504960 | elapsed time per iteration (s): 0.47 | learning rate: 4.744E-05 | global batch size:   256 | lm loss: 3.342693E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.090 | TFLOPs: 35.39 |
7:  iteration     8430/   11269 | consumed samples:      2158080 | consumed tokens:   4419747840 | elapsed time per iteration (s): 0.47 | learning rate: 4.726E-05 | global batch size:   256 | lm loss: 3.347787E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.683 | TFLOPs: 35.36 |
7:  iteration     8440/   11269 | consumed samples:      2160640 | consumed tokens:   4424990720 | elapsed time per iteration (s): 0.47 | learning rate: 4.708E-05 | global batch size:   256 | lm loss: 3.344220E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.360 | TFLOPs: 35.34 |
7:  iteration     8450/   11269 | consumed samples:      2163200 | consumed tokens:   4430233600 | elapsed time per iteration (s): 0.48 | learning rate: 4.690E-05 | global batch size:   256 | lm loss: 3.343270E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.192 | TFLOPs: 35.20 |
7:  iteration     8460/   11269 | consumed samples:      2165760 | consumed tokens:   4435476480 | elapsed time per iteration (s): 0.48 | learning rate: 4.672E-05 | global batch size:   256 | lm loss: 3.333337E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.467 | TFLOPs: 34.95 |
7:  iteration     8470/   11269 | consumed samples:      2168320 | consumed tokens:   4440719360 | elapsed time per iteration (s): 0.48 | learning rate: 4.654E-05 | global batch size:   256 | lm loss: 3.324895E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.339 | TFLOPs: 35.21 |
7:  iteration     8480/   11269 | consumed samples:      2170880 | consumed tokens:   4445962240 | elapsed time per iteration (s): 0.47 | learning rate: 4.636E-05 | global batch size:   256 | lm loss: 3.331133E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.056 | TFLOPs: 35.32 |
7:  iteration     8490/   11269 | consumed samples:      2173440 | consumed tokens:   4451205120 | elapsed time per iteration (s): 0.47 | learning rate: 4.618E-05 | global batch size:   256 | lm loss: 3.347952E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.302 | TFLOPs: 35.34 |
7:  iteration     8500/   11269 | consumed samples:      2176000 | consumed tokens:   4456448000 | elapsed time per iteration (s): 0.48 | learning rate: 4.600E-05 | global batch size:   256 | lm loss: 3.347871E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.147 | TFLOPs: 35.06 |
7:  iteration     8510/   11269 | consumed samples:      2178560 | consumed tokens:   4461690880 | elapsed time per iteration (s): 0.48 | learning rate: 4.582E-05 | global batch size:   256 | lm loss: 3.344165E+00 | grad norm: 0.286 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.975 | TFLOPs: 35.05 |
7:  iteration     8520/   11269 | consumed samples:      2181120 | consumed tokens:   4466933760 | elapsed time per iteration (s): 0.48 | learning rate: 4.565E-05 | global batch size:   256 | lm loss: 3.340998E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.721 | TFLOPs: 35.17 |
7:  iteration     8530/   11269 | consumed samples:      2183680 | consumed tokens:   4472176640 | elapsed time per iteration (s): 0.47 | learning rate: 4.547E-05 | global batch size:   256 | lm loss: 3.333259E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.059 | TFLOPs: 35.32 |
7:  iteration     8540/   11269 | consumed samples:      2186240 | consumed tokens:   4477419520 | elapsed time per iteration (s): 0.48 | learning rate: 4.529E-05 | global batch size:   256 | lm loss: 3.334092E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.024 | TFLOPs: 35.06 |
7:  iteration     8550/   11269 | consumed samples:      2188800 | consumed tokens:   4482662400 | elapsed time per iteration (s): 0.48 | learning rate: 4.512E-05 | global batch size:   256 | lm loss: 3.343022E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.228 | TFLOPs: 35.00 |
7:  iteration     8560/   11269 | consumed samples:      2191360 | consumed tokens:   4487905280 | elapsed time per iteration (s): 0.47 | learning rate: 4.494E-05 | global batch size:   256 | lm loss: 3.336051E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.028 | TFLOPs: 35.32 |
7:  iteration     8570/   11269 | consumed samples:      2193920 | consumed tokens:   4493148160 | elapsed time per iteration (s): 0.48 | learning rate: 4.477E-05 | global batch size:   256 | lm loss: 3.330336E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.049 | TFLOPs: 35.12 |
7:  iteration     8580/   11269 | consumed samples:      2196480 | consumed tokens:   4498391040 | elapsed time per iteration (s): 0.48 | learning rate: 4.459E-05 | global batch size:   256 | lm loss: 3.327454E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 531.281 | TFLOPs: 34.75 |
7:  iteration     8590/   11269 | consumed samples:      2199040 | consumed tokens:   4503633920 | elapsed time per iteration (s): 0.48 | learning rate: 4.442E-05 | global batch size:   256 | lm loss: 3.339904E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.165 | TFLOPs: 35.07 |
7:  iteration     8600/   11269 | consumed samples:      2201600 | consumed tokens:   4508876800 | elapsed time per iteration (s): 0.48 | learning rate: 4.425E-05 | global batch size:   256 | lm loss: 3.332501E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.627 | TFLOPs: 35.10 |
7:  iteration     8610/   11269 | consumed samples:      2204160 | consumed tokens:   4514119680 | elapsed time per iteration (s): 0.48 | learning rate: 4.407E-05 | global batch size:   256 | lm loss: 3.350783E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.134 | TFLOPs: 35.06 |
7:  iteration     8620/   11269 | consumed samples:      2206720 | consumed tokens:   4519362560 | elapsed time per iteration (s): 0.48 | learning rate: 4.390E-05 | global batch size:   256 | lm loss: 3.328754E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.111 | TFLOPs: 35.00 |
7:  iteration     8630/   11269 | consumed samples:      2209280 | consumed tokens:   4524605440 | elapsed time per iteration (s): 0.48 | learning rate: 4.373E-05 | global batch size:   256 | lm loss: 3.339239E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 533.707 | TFLOPs: 34.90 |
7:  iteration     8640/   11269 | consumed samples:      2211840 | consumed tokens:   4529848320 | elapsed time per iteration (s): 0.47 | learning rate: 4.356E-05 | global batch size:   256 | lm loss: 3.346037E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.203 | TFLOPs: 35.33 |
7:  iteration     8650/   11269 | consumed samples:      2214400 | consumed tokens:   4535091200 | elapsed time per iteration (s): 0.48 | learning rate: 4.339E-05 | global batch size:   256 | lm loss: 3.328410E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.374 | TFLOPs: 34.95 |
7:  iteration     8660/   11269 | consumed samples:      2216960 | consumed tokens:   4540334080 | elapsed time per iteration (s): 0.48 | learning rate: 4.322E-05 | global batch size:   256 | lm loss: 3.333214E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.774 | TFLOPs: 35.04 |
7:  iteration     8670/   11269 | consumed samples:      2219520 | consumed tokens:   4545576960 | elapsed time per iteration (s): 0.47 | learning rate: 4.305E-05 | global batch size:   256 | lm loss: 3.327511E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.293 | TFLOPs: 35.34 |
7:  iteration     8680/   11269 | consumed samples:      2222080 | consumed tokens:   4550819840 | elapsed time per iteration (s): 0.47 | learning rate: 4.288E-05 | global batch size:   256 | lm loss: 3.325333E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.935 | TFLOPs: 35.31 |
7:  iteration     8690/   11269 | consumed samples:      2224640 | consumed tokens:   4556062720 | elapsed time per iteration (s): 0.47 | learning rate: 4.271E-05 | global batch size:   256 | lm loss: 3.328397E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.859 | TFLOPs: 35.31 |
7:  iteration     8700/   11269 | consumed samples:      2227200 | consumed tokens:   4561305600 | elapsed time per iteration (s): 0.47 | learning rate: 4.254E-05 | global batch size:   256 | lm loss: 3.335538E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.595 | TFLOPs: 35.29 |
7:  iteration     8710/   11269 | consumed samples:      2229760 | consumed tokens:   4566548480 | elapsed time per iteration (s): 0.48 | learning rate: 4.237E-05 | global batch size:   256 | lm loss: 3.335431E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.943 | TFLOPs: 35.18 |
7:  iteration     8720/   11269 | consumed samples:      2232320 | consumed tokens:   4571791360 | elapsed time per iteration (s): 0.47 | learning rate: 4.221E-05 | global batch size:   256 | lm loss: 3.327683E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.992 | TFLOPs: 35.32 |
7:  iteration     8730/   11269 | consumed samples:      2234880 | consumed tokens:   4577034240 | elapsed time per iteration (s): 0.47 | learning rate: 4.204E-05 | global batch size:   256 | lm loss: 3.332487E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.147 | TFLOPs: 35.26 |
7:  iteration     8740/   11269 | consumed samples:      2237440 | consumed tokens:   4582277120 | elapsed time per iteration (s): 0.47 | learning rate: 4.188E-05 | global batch size:   256 | lm loss: 3.333670E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.724 | TFLOPs: 35.30 |
7:  iteration     8750/   11269 | consumed samples:      2240000 | consumed tokens:   4587520000 | elapsed time per iteration (s): 0.48 | learning rate: 4.171E-05 | global batch size:   256 | lm loss: 3.339067E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.573 | TFLOPs: 35.09 |
7:  iteration     8760/   11269 | consumed samples:      2242560 | consumed tokens:   4592762880 | elapsed time per iteration (s): 0.47 | learning rate: 4.154E-05 | global batch size:   256 | lm loss: 3.330478E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.943 | TFLOPs: 35.31 |
7:  iteration     8770/   11269 | consumed samples:      2245120 | consumed tokens:   4598005760 | elapsed time per iteration (s): 0.47 | learning rate: 4.138E-05 | global batch size:   256 | lm loss: 3.340259E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.858 | TFLOPs: 35.31 |
7:  iteration     8780/   11269 | consumed samples:      2247680 | consumed tokens:   4603248640 | elapsed time per iteration (s): 0.47 | learning rate: 4.122E-05 | global batch size:   256 | lm loss: 3.326702E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.819 | TFLOPs: 35.30 |
7:  iteration     8790/   11269 | consumed samples:      2250240 | consumed tokens:   4608491520 | elapsed time per iteration (s): 0.47 | learning rate: 4.105E-05 | global batch size:   256 | lm loss: 3.328143E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.728 | TFLOPs: 35.30 |
7:  iteration     8800/   11269 | consumed samples:      2252800 | consumed tokens:   4613734400 | elapsed time per iteration (s): 0.47 | learning rate: 4.089E-05 | global batch size:   256 | lm loss: 3.337477E+00 | grad norm: 0.283 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.882 | TFLOPs: 35.31 |
7:  iteration     8810/   11269 | consumed samples:      2255360 | consumed tokens:   4618977280 | elapsed time per iteration (s): 0.47 | learning rate: 4.073E-05 | global batch size:   256 | lm loss: 3.337276E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.109 | TFLOPs: 35.32 |
7:  iteration     8820/   11269 | consumed samples:      2257920 | consumed tokens:   4624220160 | elapsed time per iteration (s): 0.47 | learning rate: 4.057E-05 | global batch size:   256 | lm loss: 3.337083E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.852 | TFLOPs: 35.31 |
7:  iteration     8830/   11269 | consumed samples:      2260480 | consumed tokens:   4629463040 | elapsed time per iteration (s): 0.48 | learning rate: 4.041E-05 | global batch size:   256 | lm loss: 3.318769E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.161 | TFLOPs: 35.20 |
7:  iteration     8840/   11269 | consumed samples:      2263040 | consumed tokens:   4634705920 | elapsed time per iteration (s): 0.48 | learning rate: 4.025E-05 | global batch size:   256 | lm loss: 3.316468E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.804 | TFLOPs: 35.11 |
7:  iteration     8850/   11269 | consumed samples:      2265600 | consumed tokens:   4639948800 | elapsed time per iteration (s): 0.47 | learning rate: 4.009E-05 | global batch size:   256 | lm loss: 3.331725E+00 | grad norm: 0.286 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.082 | TFLOPs: 35.32 |
7:  iteration     8860/   11269 | consumed samples:      2268160 | consumed tokens:   4645191680 | elapsed time per iteration (s): 0.48 | learning rate: 3.993E-05 | global batch size:   256 | lm loss: 3.343082E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.697 | TFLOPs: 35.03 |
7:  iteration     8870/   11269 | consumed samples:      2270720 | consumed tokens:   4650434560 | elapsed time per iteration (s): 0.47 | learning rate: 3.977E-05 | global batch size:   256 | lm loss: 3.332667E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.962 | TFLOPs: 35.31 |
7:  iteration     8880/   11269 | consumed samples:      2273280 | consumed tokens:   4655677440 | elapsed time per iteration (s): 0.47 | learning rate: 3.961E-05 | global batch size:   256 | lm loss: 3.332137E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.761 | TFLOPs: 35.30 |
7:  iteration     8890/   11269 | consumed samples:      2275840 | consumed tokens:   4660920320 | elapsed time per iteration (s): 0.47 | learning rate: 3.945E-05 | global batch size:   256 | lm loss: 3.339101E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.953 | TFLOPs: 35.31 |
7:  iteration     8900/   11269 | consumed samples:      2278400 | consumed tokens:   4666163200 | elapsed time per iteration (s): 0.47 | learning rate: 3.930E-05 | global batch size:   256 | lm loss: 3.326251E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.815 | TFLOPs: 35.30 |
7:  iteration     8910/   11269 | consumed samples:      2280960 | consumed tokens:   4671406080 | elapsed time per iteration (s): 0.47 | learning rate: 3.914E-05 | global batch size:   256 | lm loss: 3.337200E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.160 | TFLOPs: 35.26 |
7:  iteration     8920/   11269 | consumed samples:      2283520 | consumed tokens:   4676648960 | elapsed time per iteration (s): 0.47 | learning rate: 3.898E-05 | global batch size:   256 | lm loss: 3.332147E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.476 | TFLOPs: 35.28 |
7:  iteration     8930/   11269 | consumed samples:      2286080 | consumed tokens:   4681891840 | elapsed time per iteration (s): 0.47 | learning rate: 3.883E-05 | global batch size:   256 | lm loss: 3.331721E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.498 | TFLOPs: 35.28 |
7:  iteration     8940/   11269 | consumed samples:      2288640 | consumed tokens:   4687134720 | elapsed time per iteration (s): 0.47 | learning rate: 3.867E-05 | global batch size:   256 | lm loss: 3.328350E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.681 | TFLOPs: 35.29 |
7:  iteration     8950/   11269 | consumed samples:      2291200 | consumed tokens:   4692377600 | elapsed time per iteration (s): 0.47 | learning rate: 3.852E-05 | global batch size:   256 | lm loss: 3.316887E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.582 | TFLOPs: 35.29 |
7:  iteration     8960/   11269 | consumed samples:      2293760 | consumed tokens:   4697620480 | elapsed time per iteration (s): 0.48 | learning rate: 3.836E-05 | global batch size:   256 | lm loss: 3.329145E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.130 | TFLOPs: 34.93 |
7:  iteration     8970/   11269 | consumed samples:      2296320 | consumed tokens:   4702863360 | elapsed time per iteration (s): 0.47 | learning rate: 3.821E-05 | global batch size:   256 | lm loss: 3.328048E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.220 | TFLOPs: 35.26 |
7:  iteration     8980/   11269 | consumed samples:      2298880 | consumed tokens:   4708106240 | elapsed time per iteration (s): 0.47 | learning rate: 3.806E-05 | global batch size:   256 | lm loss: 3.324206E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.362 | TFLOPs: 35.27 |
7:  iteration     8990/   11269 | consumed samples:      2301440 | consumed tokens:   4713349120 | elapsed time per iteration (s): 0.47 | learning rate: 3.791E-05 | global batch size:   256 | lm loss: 3.333245E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.430 | TFLOPs: 35.28 |
7:  iteration     9000/   11269 | consumed samples:      2304000 | consumed tokens:   4718592000 | elapsed time per iteration (s): 0.47 | learning rate: 3.776E-05 | global batch size:   256 | lm loss: 3.321058E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.735 | TFLOPs: 35.30 |
7: -----------------------------------------------------------------------------------------------
7: validation loss at iteration 9000 | lm loss value: 3.392526E+00 | lm loss PPL: 2.974097E+01 | 
7: -----------------------------------------------------------------------------------------------
0: saving checkpoint at iteration    9000 to checkpoints_280m5b9400m
0: [2023-03-15 23:11:17,206] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step9000 is begin to save!
0: [2023-03-15 23:11:17,211] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_01-model_00-model_states.pt...
0: [2023-03-15 23:11:17,352] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_01-model_00-model_states.pt.
0: [2023-03-15 23:11:17,352] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_03-model_00-model_states.pt...
0: [2023-03-15 23:11:17,378] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_03-model_00-model_states.pt.
0: [2023-03-15 23:11:17,378] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_04-model_00-model_states.pt...
0: [2023-03-15 23:11:17,403] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_04-model_00-model_states.pt.
0: [2023-03-15 23:11:17,404] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_05-model_00-model_states.pt...
0: [2023-03-15 23:11:17,428] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_05-model_00-model_states.pt.
0: [2023-03-15 23:11:17,429] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_06-model_00-model_states.pt...
0: [2023-03-15 23:11:17,454] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_06-model_00-model_states.pt.
0: [2023-03-15 23:11:17,454] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_07-model_00-model_states.pt...
0: [2023-03-15 23:11:17,479] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_07-model_00-model_states.pt.
0: [2023-03-15 23:11:17,479] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_08-model_00-model_states.pt...
0: [2023-03-15 23:11:17,504] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_08-model_00-model_states.pt.
0: [2023-03-15 23:11:17,505] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_09-model_00-model_states.pt...
0: [2023-03-15 23:11:17,529] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_09-model_00-model_states.pt.
0: [2023-03-15 23:11:17,530] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_10-model_00-model_states.pt...
0: [2023-03-15 23:11:17,554] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_10-model_00-model_states.pt.
0: [2023-03-15 23:11:17,555] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_11-model_00-model_states.pt...
0: [2023-03-15 23:11:17,579] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_11-model_00-model_states.pt.
0: [2023-03-15 23:11:17,580] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_12-model_00-model_states.pt...
0: [2023-03-15 23:11:17,604] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_12-model_00-model_states.pt.
0: [2023-03-15 23:11:17,604] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_13-model_00-model_states.pt...
0: [2023-03-15 23:11:17,629] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_13-model_00-model_states.pt.
0: [2023-03-15 23:11:17,629] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_14-model_00-model_states.pt...
0: [2023-03-15 23:11:17,654] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_14-model_00-model_states.pt.
0: [2023-03-15 23:11:17,654] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_15-model_00-model_states.pt...
0: [2023-03-15 23:11:17,679] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_15-model_00-model_states.pt.
0: [2023-03-15 23:11:17,679] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_16-model_00-model_states.pt...
0: [2023-03-15 23:11:17,705] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_16-model_00-model_states.pt.
0: [2023-03-15 23:11:17,705] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_17-model_00-model_states.pt...
0: [2023-03-15 23:11:17,730] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_17-model_00-model_states.pt.
0: [2023-03-15 23:11:17,730] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_18-model_00-model_states.pt...
0: [2023-03-15 23:11:17,755] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_18-model_00-model_states.pt.
0: [2023-03-15 23:11:17,755] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_19-model_00-model_states.pt...
0: [2023-03-15 23:11:17,780] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_19-model_00-model_states.pt.
0: [2023-03-15 23:11:17,780] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_20-model_00-model_states.pt...
0: [2023-03-15 23:11:17,804] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_20-model_00-model_states.pt.
0: [2023-03-15 23:11:17,804] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/layer_22-model_00-model_states.pt...
0: [2023-03-15 23:11:17,805] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/layer_22-model_00-model_states.pt.
0: [2023-03-15 23:11:17,806] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_280m5b9400m/global_step9000/mp_rank_00_model_states.pt
0: [2023-03-15 23:11:17,806] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/mp_rank_00_model_states.pt...
0: [2023-03-15 23:11:17,809] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/mp_rank_00_model_states.pt.
0: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:11:17,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:11:17,919] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:11:17,919] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:11:17,919] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:11:17,919] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:11:17,919] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-15 23:11:17,919] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-15 23:11:17,919] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-15 23:11:17,919] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-15 23:11:17,919] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
7: [2023-03-15 23:11:17,919] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
7: [2023-03-15 23:11:17,919] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
7: [2023-03-15 23:11:17,919] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
7: [2023-03-15 23:11:17,920] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:11:17,920] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-15 23:11:17,920] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
7: [2023-03-15 23:11:17,920] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:11:17,920] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-15 23:11:17,920] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
7: [2023-03-15 23:11:17,921] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:11:17,921] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-15 23:11:17,921] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
7: [2023-03-15 23:11:17,921] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:11:17,922] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-15 23:11:17,922] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
0: [2023-03-15 23:11:17,935] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:11:17,935] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:11:17,935] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-15 23:11:17,935] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-15 23:11:17,935] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
0: [2023-03-15 23:11:17,935] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
0: [2023-03-15 23:11:17,935] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:11:17,935] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-15 23:11:17,935] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
0: [2023-03-15 23:11:17,935] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:11:17,936] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-15 23:11:17,936] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
0: [2023-03-15 23:11:17,949] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:11:17,949] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-15 23:11:17,949] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:11:17,949] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
0: [2023-03-15 23:11:17,949] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:11:17,949] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-15 23:11:17,949] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
0: [2023-03-15 23:11:17,949] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:11:17,949] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-15 23:11:17,949] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
6: [2023-03-15 23:11:17,956] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:11:17,956] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:11:17,956] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:11:17,956] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:11:17,956] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:11:17,956] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:11:17,956] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:11:17,956] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-15 23:11:17,956] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-15 23:11:17,956] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-15 23:11:17,956] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-15 23:11:17,956] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-15 23:11:17,956] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-15 23:11:17,956] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
6: [2023-03-15 23:11:17,956] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
6: [2023-03-15 23:11:17,956] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-15 23:11:17,956] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:11:17,956] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
6: [2023-03-15 23:11:17,956] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
6: [2023-03-15 23:11:17,956] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
6: [2023-03-15 23:11:17,956] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
6: [2023-03-15 23:11:17,956] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
6: [2023-03-15 23:11:17,956] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-15 23:11:17,956] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
5: [2023-03-15 23:11:17,967] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:11:17,967] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:11:17,967] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:11:17,967] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:11:17,967] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:11:17,967] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:11:17,967] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:11:17,967] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-15 23:11:17,967] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-15 23:11:17,967] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-15 23:11:17,967] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-15 23:11:17,967] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:11:17,967] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-15 23:11:17,967] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-15 23:11:17,967] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-15 23:11:17,967] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
5: [2023-03-15 23:11:17,967] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
5: [2023-03-15 23:11:17,967] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
5: [2023-03-15 23:11:17,967] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
5: [2023-03-15 23:11:17,967] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-15 23:11:17,967] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
5: [2023-03-15 23:11:17,967] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
5: [2023-03-15 23:11:17,967] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
5: [2023-03-15 23:11:17,967] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
2: [2023-03-15 23:11:17,990] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:11:17,990] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:11:17,990] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:11:17,990] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:11:17,990] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:11:17,990] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:11:17,990] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:11:17,990] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:11:17,990] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-15 23:11:17,990] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-15 23:11:17,990] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-15 23:11:17,990] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-15 23:11:17,990] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-15 23:11:17,990] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-15 23:11:17,990] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-15 23:11:17,990] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-15 23:11:17,990] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
2: [2023-03-15 23:11:17,990] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
2: [2023-03-15 23:11:17,990] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
2: [2023-03-15 23:11:17,990] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
2: [2023-03-15 23:11:17,990] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
2: [2023-03-15 23:11:17,990] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
2: [2023-03-15 23:11:17,990] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
2: [2023-03-15 23:11:17,990] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
0: [2023-03-15 23:11:17,991] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-15 23:11:17,991] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
4: [2023-03-15 23:11:17,992] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:11:17,992] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:11:17,992] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:11:17,992] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:11:17,992] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:11:17,992] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:11:17,992] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:11:17,992] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-15 23:11:17,992] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-15 23:11:17,992] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-15 23:11:17,992] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-15 23:11:17,992] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
4: [2023-03-15 23:11:17,992] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-15 23:11:17,992] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-15 23:11:17,992] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-15 23:11:17,992] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
4: [2023-03-15 23:11:17,992] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
4: [2023-03-15 23:11:17,992] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
4: [2023-03-15 23:11:17,992] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
4: [2023-03-15 23:11:17,992] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
4: [2023-03-15 23:11:17,992] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
4: [2023-03-15 23:11:17,999] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:11:17,999] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-15 23:11:17,999] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
1: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:11:18,003] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-15 23:11:18,003] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-15 23:11:18,003] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-15 23:11:18,003] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-15 23:11:18,003] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
3: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:11:18,003] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
3: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:11:18,003] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-15 23:11:18,003] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
3: [2023-03-15 23:11:18,004] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-15 23:11:18,004] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
1: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
1: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
1: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
1: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
3: [2023-03-15 23:11:18,004] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-15 23:11:18,004] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-15 23:11:18,004] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-15 23:11:18,004] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-15 23:11:18,004] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
1: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
1: [2023-03-15 23:11:18,003] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
3: [2023-03-15 23:11:18,004] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step9000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-15 23:11:18,004] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
3: [2023-03-15 23:11:18,004] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
1: [2023-03-15 23:11:18,004] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
1: [2023-03-15 23:11:18,004] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
3: [2023-03-15 23:11:18,004] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
3: [2023-03-15 23:11:18,004] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
3: [2023-03-15 23:11:18,004] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
3: [2023-03-15 23:11:18,004] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
3: [2023-03-15 23:11:18,004] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
3: [2023-03-15 23:11:18,004] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step9000 is ready now!
0:   successfully saved checkpoint at iteration    9000 to checkpoints_280m5b9400m
7: time (ms) | save-checkpoint: 848.71
7:  iteration     9010/   11269 | consumed samples:      2306560 | consumed tokens:   4723834880 | elapsed time per iteration (s): 0.57 | learning rate: 3.760E-05 | global batch size:   256 | lm loss: 3.325890E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 447.328 | TFLOPs: 29.26 |
7:  iteration     9020/   11269 | consumed samples:      2309120 | consumed tokens:   4729077760 | elapsed time per iteration (s): 0.47 | learning rate: 3.745E-05 | global batch size:   256 | lm loss: 3.316447E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.624 | TFLOPs: 35.42 |
7:  iteration     9030/   11269 | consumed samples:      2311680 | consumed tokens:   4734320640 | elapsed time per iteration (s): 0.47 | learning rate: 3.730E-05 | global batch size:   256 | lm loss: 3.320748E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.124 | TFLOPs: 35.39 |
7:  iteration     9040/   11269 | consumed samples:      2314240 | consumed tokens:   4739563520 | elapsed time per iteration (s): 0.47 | learning rate: 3.716E-05 | global batch size:   256 | lm loss: 3.325248E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.468 | TFLOPs: 35.35 |
7:  iteration     9050/   11269 | consumed samples:      2316800 | consumed tokens:   4744806400 | elapsed time per iteration (s): 0.47 | learning rate: 3.701E-05 | global batch size:   256 | lm loss: 3.322734E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.300 | TFLOPs: 35.34 |
7:  iteration     9060/   11269 | consumed samples:      2319360 | consumed tokens:   4750049280 | elapsed time per iteration (s): 0.47 | learning rate: 3.686E-05 | global batch size:   256 | lm loss: 3.338685E+00 | grad norm: 0.286 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.288 | TFLOPs: 35.33 |
7:  iteration     9070/   11269 | consumed samples:      2321920 | consumed tokens:   4755292160 | elapsed time per iteration (s): 0.47 | learning rate: 3.671E-05 | global batch size:   256 | lm loss: 3.330685E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.723 | TFLOPs: 35.30 |
7:  iteration     9080/   11269 | consumed samples:      2324480 | consumed tokens:   4760535040 | elapsed time per iteration (s): 0.47 | learning rate: 3.656E-05 | global batch size:   256 | lm loss: 3.333285E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.508 | TFLOPs: 35.28 |
7:  iteration     9090/   11269 | consumed samples:      2327040 | consumed tokens:   4765777920 | elapsed time per iteration (s): 0.47 | learning rate: 3.642E-05 | global batch size:   256 | lm loss: 3.328292E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.979 | TFLOPs: 35.31 |
7:  iteration     9100/   11269 | consumed samples:      2329600 | consumed tokens:   4771020800 | elapsed time per iteration (s): 0.47 | learning rate: 3.627E-05 | global batch size:   256 | lm loss: 3.320432E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.745 | TFLOPs: 35.30 |
7:  iteration     9110/   11269 | consumed samples:      2332160 | consumed tokens:   4776263680 | elapsed time per iteration (s): 0.47 | learning rate: 3.613E-05 | global batch size:   256 | lm loss: 3.325027E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.713 | TFLOPs: 35.30 |
7:  iteration     9120/   11269 | consumed samples:      2334720 | consumed tokens:   4781506560 | elapsed time per iteration (s): 0.47 | learning rate: 3.598E-05 | global batch size:   256 | lm loss: 3.321106E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.780 | TFLOPs: 35.30 |
7:  iteration     9130/   11269 | consumed samples:      2337280 | consumed tokens:   4786749440 | elapsed time per iteration (s): 0.47 | learning rate: 3.584E-05 | global batch size:   256 | lm loss: 3.317316E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.928 | TFLOPs: 35.31 |
7:  iteration     9140/   11269 | consumed samples:      2339840 | consumed tokens:   4791992320 | elapsed time per iteration (s): 0.47 | learning rate: 3.570E-05 | global batch size:   256 | lm loss: 3.326775E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.954 | TFLOPs: 35.31 |
7:  iteration     9150/   11269 | consumed samples:      2342400 | consumed tokens:   4797235200 | elapsed time per iteration (s): 0.47 | learning rate: 3.555E-05 | global batch size:   256 | lm loss: 3.335121E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.198 | TFLOPs: 35.33 |
7:  iteration     9160/   11269 | consumed samples:      2344960 | consumed tokens:   4802478080 | elapsed time per iteration (s): 0.47 | learning rate: 3.541E-05 | global batch size:   256 | lm loss: 3.328239E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.836 | TFLOPs: 35.31 |
7:  iteration     9170/   11269 | consumed samples:      2347520 | consumed tokens:   4807720960 | elapsed time per iteration (s): 0.47 | learning rate: 3.527E-05 | global batch size:   256 | lm loss: 3.339150E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.919 | TFLOPs: 35.31 |
7:  iteration     9180/   11269 | consumed samples:      2350080 | consumed tokens:   4812963840 | elapsed time per iteration (s): 0.47 | learning rate: 3.513E-05 | global batch size:   256 | lm loss: 3.332687E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.006 | TFLOPs: 35.32 |
7:  iteration     9190/   11269 | consumed samples:      2352640 | consumed tokens:   4818206720 | elapsed time per iteration (s): 0.47 | learning rate: 3.499E-05 | global batch size:   256 | lm loss: 3.323779E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.600 | TFLOPs: 35.29 |
7:  iteration     9200/   11269 | consumed samples:      2355200 | consumed tokens:   4823449600 | elapsed time per iteration (s): 0.47 | learning rate: 3.485E-05 | global batch size:   256 | lm loss: 3.335056E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.814 | TFLOPs: 35.30 |
7:  iteration     9210/   11269 | consumed samples:      2357760 | consumed tokens:   4828692480 | elapsed time per iteration (s): 0.47 | learning rate: 3.471E-05 | global batch size:   256 | lm loss: 3.330942E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.592 | TFLOPs: 35.29 |
7:  iteration     9220/   11269 | consumed samples:      2360320 | consumed tokens:   4833935360 | elapsed time per iteration (s): 0.48 | learning rate: 3.457E-05 | global batch size:   256 | lm loss: 3.322591E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.738 | TFLOPs: 35.10 |
7:  iteration     9230/   11269 | consumed samples:      2362880 | consumed tokens:   4839178240 | elapsed time per iteration (s): 0.47 | learning rate: 3.443E-05 | global batch size:   256 | lm loss: 3.307954E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.182 | TFLOPs: 35.33 |
7:  iteration     9240/   11269 | consumed samples:      2365440 | consumed tokens:   4844421120 | elapsed time per iteration (s): 0.47 | learning rate: 3.430E-05 | global batch size:   256 | lm loss: 3.319587E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.232 | TFLOPs: 35.33 |
7:  iteration     9250/   11269 | consumed samples:      2368000 | consumed tokens:   4849664000 | elapsed time per iteration (s): 0.47 | learning rate: 3.416E-05 | global batch size:   256 | lm loss: 3.321380E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.117 | TFLOPs: 35.32 |
7:  iteration     9260/   11269 | consumed samples:      2370560 | consumed tokens:   4854906880 | elapsed time per iteration (s): 0.48 | learning rate: 3.402E-05 | global batch size:   256 | lm loss: 3.331778E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.225 | TFLOPs: 35.07 |
7:  iteration     9270/   11269 | consumed samples:      2373120 | consumed tokens:   4860149760 | elapsed time per iteration (s): 0.47 | learning rate: 3.389E-05 | global batch size:   256 | lm loss: 3.316853E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.275 | TFLOPs: 35.33 |
7:  iteration     9280/   11269 | consumed samples:      2375680 | consumed tokens:   4865392640 | elapsed time per iteration (s): 0.47 | learning rate: 3.375E-05 | global batch size:   256 | lm loss: 3.312777E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.732 | TFLOPs: 35.30 |
7:  iteration     9290/   11269 | consumed samples:      2378240 | consumed tokens:   4870635520 | elapsed time per iteration (s): 0.48 | learning rate: 3.362E-05 | global batch size:   256 | lm loss: 3.320986E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.549 | TFLOPs: 35.22 |
7:  iteration     9300/   11269 | consumed samples:      2380800 | consumed tokens:   4875878400 | elapsed time per iteration (s): 0.47 | learning rate: 3.348E-05 | global batch size:   256 | lm loss: 3.325394E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.919 | TFLOPs: 35.31 |
7:  iteration     9310/   11269 | consumed samples:      2383360 | consumed tokens:   4881121280 | elapsed time per iteration (s): 0.48 | learning rate: 3.335E-05 | global batch size:   256 | lm loss: 3.328994E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.318 | TFLOPs: 35.01 |
7:  iteration     9320/   11269 | consumed samples:      2385920 | consumed tokens:   4886364160 | elapsed time per iteration (s): 0.47 | learning rate: 3.322E-05 | global batch size:   256 | lm loss: 3.314394E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.277 | TFLOPs: 35.33 |
7:  iteration     9330/   11269 | consumed samples:      2388480 | consumed tokens:   4891607040 | elapsed time per iteration (s): 0.47 | learning rate: 3.309E-05 | global batch size:   256 | lm loss: 3.320523E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.491 | TFLOPs: 35.35 |
7:  iteration     9340/   11269 | consumed samples:      2391040 | consumed tokens:   4896849920 | elapsed time per iteration (s): 0.47 | learning rate: 3.296E-05 | global batch size:   256 | lm loss: 3.312473E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.311 | TFLOPs: 35.34 |
7:  iteration     9350/   11269 | consumed samples:      2393600 | consumed tokens:   4902092800 | elapsed time per iteration (s): 0.47 | learning rate: 3.282E-05 | global batch size:   256 | lm loss: 3.311703E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.031 | TFLOPs: 35.32 |
7:  iteration     9360/   11269 | consumed samples:      2396160 | consumed tokens:   4907335680 | elapsed time per iteration (s): 0.47 | learning rate: 3.269E-05 | global batch size:   256 | lm loss: 3.318692E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.238 | TFLOPs: 35.33 |
7:  iteration     9370/   11269 | consumed samples:      2398720 | consumed tokens:   4912578560 | elapsed time per iteration (s): 0.47 | learning rate: 3.257E-05 | global batch size:   256 | lm loss: 3.315847E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.601 | TFLOPs: 35.29 |
7:  iteration     9380/   11269 | consumed samples:      2401280 | consumed tokens:   4917821440 | elapsed time per iteration (s): 0.48 | learning rate: 3.244E-05 | global batch size:   256 | lm loss: 3.315657E+00 | grad norm: 0.285 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 535.824 | TFLOPs: 35.04 |
7:  iteration     9390/   11269 | consumed samples:      2403840 | consumed tokens:   4923064320 | elapsed time per iteration (s): 0.47 | learning rate: 3.231E-05 | global batch size:   256 | lm loss: 3.325801E+00 | grad norm: 0.283 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.583 | TFLOPs: 35.29 |
7:  iteration     9400/   11269 | consumed samples:      2406400 | consumed tokens:   4928307200 | elapsed time per iteration (s): 0.47 | learning rate: 3.218E-05 | global batch size:   256 | lm loss: 3.320361E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.315 | TFLOPs: 35.34 |
7:  iteration     9410/   11269 | consumed samples:      2408960 | consumed tokens:   4933550080 | elapsed time per iteration (s): 0.47 | learning rate: 3.205E-05 | global batch size:   256 | lm loss: 3.319061E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.525 | TFLOPs: 35.35 |
7:  iteration     9420/   11269 | consumed samples:      2411520 | consumed tokens:   4938792960 | elapsed time per iteration (s): 0.47 | learning rate: 3.193E-05 | global batch size:   256 | lm loss: 3.311233E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.997 | TFLOPs: 35.32 |
7:  iteration     9430/   11269 | consumed samples:      2414080 | consumed tokens:   4944035840 | elapsed time per iteration (s): 0.48 | learning rate: 3.180E-05 | global batch size:   256 | lm loss: 3.318502E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.456 | TFLOPs: 35.15 |
7:  iteration     9440/   11269 | consumed samples:      2416640 | consumed tokens:   4949278720 | elapsed time per iteration (s): 0.48 | learning rate: 3.168E-05 | global batch size:   256 | lm loss: 3.324685E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 528.973 | TFLOPs: 34.59 |
7:  iteration     9450/   11269 | consumed samples:      2419200 | consumed tokens:   4954521600 | elapsed time per iteration (s): 0.48 | learning rate: 3.155E-05 | global batch size:   256 | lm loss: 3.323498E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 527.922 | TFLOPs: 34.53 |
7:  iteration     9460/   11269 | consumed samples:      2421760 | consumed tokens:   4959764480 | elapsed time per iteration (s): 0.48 | learning rate: 3.143E-05 | global batch size:   256 | lm loss: 3.330342E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 532.370 | TFLOPs: 34.82 |
7:  iteration     9470/   11269 | consumed samples:      2424320 | consumed tokens:   4965007360 | elapsed time per iteration (s): 0.48 | learning rate: 3.130E-05 | global batch size:   256 | lm loss: 3.315230E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 534.502 | TFLOPs: 34.96 |
7:  iteration     9480/   11269 | consumed samples:      2426880 | consumed tokens:   4970250240 | elapsed time per iteration (s): 0.48 | learning rate: 3.118E-05 | global batch size:   256 | lm loss: 3.321272E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.332 | TFLOPs: 35.08 |
7:  iteration     9490/   11269 | consumed samples:      2429440 | consumed tokens:   4975493120 | elapsed time per iteration (s): 0.48 | learning rate: 3.106E-05 | global batch size:   256 | lm loss: 3.335855E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.651 | TFLOPs: 35.23 |
7:  iteration     9500/   11269 | consumed samples:      2432000 | consumed tokens:   4980736000 | elapsed time per iteration (s): 0.47 | learning rate: 3.094E-05 | global batch size:   256 | lm loss: 3.311408E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.716 | TFLOPs: 35.30 |
7:  iteration     9510/   11269 | consumed samples:      2434560 | consumed tokens:   4985978880 | elapsed time per iteration (s): 0.47 | learning rate: 3.082E-05 | global batch size:   256 | lm loss: 3.313414E+00 | grad norm: 0.286 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.828 | TFLOPs: 35.30 |
7:  iteration     9520/   11269 | consumed samples:      2437120 | consumed tokens:   4991221760 | elapsed time per iteration (s): 0.47 | learning rate: 3.070E-05 | global batch size:   256 | lm loss: 3.314258E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.128 | TFLOPs: 35.32 |
7:  iteration     9530/   11269 | consumed samples:      2439680 | consumed tokens:   4996464640 | elapsed time per iteration (s): 0.47 | learning rate: 3.058E-05 | global batch size:   256 | lm loss: 3.315903E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.130 | TFLOPs: 35.32 |
7:  iteration     9540/   11269 | consumed samples:      2442240 | consumed tokens:   5001707520 | elapsed time per iteration (s): 0.47 | learning rate: 3.046E-05 | global batch size:   256 | lm loss: 3.314369E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.052 | TFLOPs: 35.32 |
7:  iteration     9550/   11269 | consumed samples:      2444800 | consumed tokens:   5006950400 | elapsed time per iteration (s): 0.47 | learning rate: 3.034E-05 | global batch size:   256 | lm loss: 3.319090E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.473 | TFLOPs: 35.35 |
7:  iteration     9560/   11269 | consumed samples:      2447360 | consumed tokens:   5012193280 | elapsed time per iteration (s): 0.47 | learning rate: 3.022E-05 | global batch size:   256 | lm loss: 3.328969E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.244 | TFLOPs: 35.33 |
7:  iteration     9570/   11269 | consumed samples:      2449920 | consumed tokens:   5017436160 | elapsed time per iteration (s): 0.47 | learning rate: 3.011E-05 | global batch size:   256 | lm loss: 3.335327E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.051 | TFLOPs: 35.32 |
7:  iteration     9580/   11269 | consumed samples:      2452480 | consumed tokens:   5022679040 | elapsed time per iteration (s): 0.47 | learning rate: 2.999E-05 | global batch size:   256 | lm loss: 3.310271E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.970 | TFLOPs: 35.31 |
7:  iteration     9590/   11269 | consumed samples:      2455040 | consumed tokens:   5027921920 | elapsed time per iteration (s): 0.47 | learning rate: 2.987E-05 | global batch size:   256 | lm loss: 3.323361E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.380 | TFLOPs: 35.34 |
7:  iteration     9600/   11269 | consumed samples:      2457600 | consumed tokens:   5033164800 | elapsed time per iteration (s): 0.47 | learning rate: 2.976E-05 | global batch size:   256 | lm loss: 3.317080E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.241 | TFLOPs: 35.33 |
7:  iteration     9610/   11269 | consumed samples:      2460160 | consumed tokens:   5038407680 | elapsed time per iteration (s): 0.47 | learning rate: 2.964E-05 | global batch size:   256 | lm loss: 3.323788E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.673 | TFLOPs: 35.29 |
7:  iteration     9620/   11269 | consumed samples:      2462720 | consumed tokens:   5043650560 | elapsed time per iteration (s): 0.47 | learning rate: 2.953E-05 | global batch size:   256 | lm loss: 3.322927E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.278 | TFLOPs: 35.33 |
7:  iteration     9630/   11269 | consumed samples:      2465280 | consumed tokens:   5048893440 | elapsed time per iteration (s): 0.47 | learning rate: 2.942E-05 | global batch size:   256 | lm loss: 3.315225E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.307 | TFLOPs: 35.34 |
7:  iteration     9640/   11269 | consumed samples:      2467840 | consumed tokens:   5054136320 | elapsed time per iteration (s): 0.47 | learning rate: 2.930E-05 | global batch size:   256 | lm loss: 3.308345E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.843 | TFLOPs: 35.31 |
7:  iteration     9650/   11269 | consumed samples:      2470400 | consumed tokens:   5059379200 | elapsed time per iteration (s): 0.47 | learning rate: 2.919E-05 | global batch size:   256 | lm loss: 3.323321E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.283 | TFLOPs: 35.33 |
7:  iteration     9660/   11269 | consumed samples:      2472960 | consumed tokens:   5064622080 | elapsed time per iteration (s): 0.47 | learning rate: 2.908E-05 | global batch size:   256 | lm loss: 3.310519E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.433 | TFLOPs: 35.34 |
7:  iteration     9670/   11269 | consumed samples:      2475520 | consumed tokens:   5069864960 | elapsed time per iteration (s): 0.47 | learning rate: 2.897E-05 | global batch size:   256 | lm loss: 3.305755E+00 | grad norm: 0.282 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.448 | TFLOPs: 35.35 |
7:  iteration     9680/   11269 | consumed samples:      2478080 | consumed tokens:   5075107840 | elapsed time per iteration (s): 0.47 | learning rate: 2.886E-05 | global batch size:   256 | lm loss: 3.312355E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.416 | TFLOPs: 35.34 |
7:  iteration     9690/   11269 | consumed samples:      2480640 | consumed tokens:   5080350720 | elapsed time per iteration (s): 0.47 | learning rate: 2.875E-05 | global batch size:   256 | lm loss: 3.316987E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.462 | TFLOPs: 35.35 |
7:  iteration     9700/   11269 | consumed samples:      2483200 | consumed tokens:   5085593600 | elapsed time per iteration (s): 0.47 | learning rate: 2.864E-05 | global batch size:   256 | lm loss: 3.317546E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.439 | TFLOPs: 35.34 |
7:  iteration     9710/   11269 | consumed samples:      2485760 | consumed tokens:   5090836480 | elapsed time per iteration (s): 0.47 | learning rate: 2.853E-05 | global batch size:   256 | lm loss: 3.310954E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.673 | TFLOPs: 35.36 |
7:  iteration     9720/   11269 | consumed samples:      2488320 | consumed tokens:   5096079360 | elapsed time per iteration (s): 0.47 | learning rate: 2.843E-05 | global batch size:   256 | lm loss: 3.320952E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.869 | TFLOPs: 35.31 |
7:  iteration     9730/   11269 | consumed samples:      2490880 | consumed tokens:   5101322240 | elapsed time per iteration (s): 0.47 | learning rate: 2.832E-05 | global batch size:   256 | lm loss: 3.312064E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.605 | TFLOPs: 35.36 |
7:  iteration     9740/   11269 | consumed samples:      2493440 | consumed tokens:   5106565120 | elapsed time per iteration (s): 0.47 | learning rate: 2.821E-05 | global batch size:   256 | lm loss: 3.307042E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.662 | TFLOPs: 35.36 |
7:  iteration     9750/   11269 | consumed samples:      2496000 | consumed tokens:   5111808000 | elapsed time per iteration (s): 0.47 | learning rate: 2.811E-05 | global batch size:   256 | lm loss: 3.302326E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.397 | TFLOPs: 35.34 |
7:  iteration     9760/   11269 | consumed samples:      2498560 | consumed tokens:   5117050880 | elapsed time per iteration (s): 0.47 | learning rate: 2.800E-05 | global batch size:   256 | lm loss: 3.320468E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.902 | TFLOPs: 35.31 |
7:  iteration     9770/   11269 | consumed samples:      2501120 | consumed tokens:   5122293760 | elapsed time per iteration (s): 0.47 | learning rate: 2.790E-05 | global batch size:   256 | lm loss: 3.316857E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.034 | TFLOPs: 35.32 |
7:  iteration     9780/   11269 | consumed samples:      2503680 | consumed tokens:   5127536640 | elapsed time per iteration (s): 0.47 | learning rate: 2.780E-05 | global batch size:   256 | lm loss: 3.309514E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.282 | TFLOPs: 35.33 |
7:  iteration     9790/   11269 | consumed samples:      2506240 | consumed tokens:   5132779520 | elapsed time per iteration (s): 0.47 | learning rate: 2.769E-05 | global batch size:   256 | lm loss: 3.310839E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.393 | TFLOPs: 35.34 |
7:  iteration     9800/   11269 | consumed samples:      2508800 | consumed tokens:   5138022400 | elapsed time per iteration (s): 0.47 | learning rate: 2.759E-05 | global batch size:   256 | lm loss: 3.309772E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.368 | TFLOPs: 35.34 |
7:  iteration     9810/   11269 | consumed samples:      2511360 | consumed tokens:   5143265280 | elapsed time per iteration (s): 0.47 | learning rate: 2.749E-05 | global batch size:   256 | lm loss: 3.312812E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.203 | TFLOPs: 35.33 |
7:  iteration     9820/   11269 | consumed samples:      2513920 | consumed tokens:   5148508160 | elapsed time per iteration (s): 0.47 | learning rate: 2.739E-05 | global batch size:   256 | lm loss: 3.314819E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.346 | TFLOPs: 35.34 |
7:  iteration     9830/   11269 | consumed samples:      2516480 | consumed tokens:   5153751040 | elapsed time per iteration (s): 0.47 | learning rate: 2.729E-05 | global batch size:   256 | lm loss: 3.305862E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.196 | TFLOPs: 35.33 |
7:  iteration     9840/   11269 | consumed samples:      2519040 | consumed tokens:   5158993920 | elapsed time per iteration (s): 0.47 | learning rate: 2.719E-05 | global batch size:   256 | lm loss: 3.300914E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.095 | TFLOPs: 35.32 |
7:  iteration     9850/   11269 | consumed samples:      2521600 | consumed tokens:   5164236800 | elapsed time per iteration (s): 0.47 | learning rate: 2.709E-05 | global batch size:   256 | lm loss: 3.316385E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.340 | TFLOPs: 35.34 |
7:  iteration     9860/   11269 | consumed samples:      2524160 | consumed tokens:   5169479680 | elapsed time per iteration (s): 0.47 | learning rate: 2.699E-05 | global batch size:   256 | lm loss: 3.305336E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.227 | TFLOPs: 35.33 |
7:  iteration     9870/   11269 | consumed samples:      2526720 | consumed tokens:   5174722560 | elapsed time per iteration (s): 0.47 | learning rate: 2.689E-05 | global batch size:   256 | lm loss: 3.317137E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.412 | TFLOPs: 35.34 |
7:  iteration     9880/   11269 | consumed samples:      2529280 | consumed tokens:   5179965440 | elapsed time per iteration (s): 0.47 | learning rate: 2.680E-05 | global batch size:   256 | lm loss: 3.328242E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.533 | TFLOPs: 35.35 |
7:  iteration     9890/   11269 | consumed samples:      2531840 | consumed tokens:   5185208320 | elapsed time per iteration (s): 0.47 | learning rate: 2.670E-05 | global batch size:   256 | lm loss: 3.309160E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.465 | TFLOPs: 35.35 |
7:  iteration     9900/   11269 | consumed samples:      2534400 | consumed tokens:   5190451200 | elapsed time per iteration (s): 0.47 | learning rate: 2.661E-05 | global batch size:   256 | lm loss: 3.303590E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.417 | TFLOPs: 35.34 |
7:  iteration     9910/   11269 | consumed samples:      2536960 | consumed tokens:   5195694080 | elapsed time per iteration (s): 0.47 | learning rate: 2.651E-05 | global batch size:   256 | lm loss: 3.306400E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.239 | TFLOPs: 35.33 |
7:  iteration     9920/   11269 | consumed samples:      2539520 | consumed tokens:   5200936960 | elapsed time per iteration (s): 0.47 | learning rate: 2.642E-05 | global batch size:   256 | lm loss: 3.303472E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.690 | TFLOPs: 35.36 |
7:  iteration     9930/   11269 | consumed samples:      2542080 | consumed tokens:   5206179840 | elapsed time per iteration (s): 0.47 | learning rate: 2.632E-05 | global batch size:   256 | lm loss: 3.315769E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.885 | TFLOPs: 35.37 |
7:  iteration     9940/   11269 | consumed samples:      2544640 | consumed tokens:   5211422720 | elapsed time per iteration (s): 0.47 | learning rate: 2.623E-05 | global batch size:   256 | lm loss: 3.318786E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.731 | TFLOPs: 35.36 |
7:  iteration     9950/   11269 | consumed samples:      2547200 | consumed tokens:   5216665600 | elapsed time per iteration (s): 0.47 | learning rate: 2.614E-05 | global batch size:   256 | lm loss: 3.295103E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.573 | TFLOPs: 35.35 |
7:  iteration     9960/   11269 | consumed samples:      2549760 | consumed tokens:   5221908480 | elapsed time per iteration (s): 0.47 | learning rate: 2.605E-05 | global batch size:   256 | lm loss: 3.300507E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.679 | TFLOPs: 35.36 |
7:  iteration     9970/   11269 | consumed samples:      2552320 | consumed tokens:   5227151360 | elapsed time per iteration (s): 0.47 | learning rate: 2.595E-05 | global batch size:   256 | lm loss: 3.303519E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.823 | TFLOPs: 35.37 |
7:  iteration     9980/   11269 | consumed samples:      2554880 | consumed tokens:   5232394240 | elapsed time per iteration (s): 0.47 | learning rate: 2.586E-05 | global batch size:   256 | lm loss: 3.304668E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.830 | TFLOPs: 35.37 |
7:  iteration     9990/   11269 | consumed samples:      2557440 | consumed tokens:   5237637120 | elapsed time per iteration (s): 0.47 | learning rate: 2.577E-05 | global batch size:   256 | lm loss: 3.317606E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.852 | TFLOPs: 35.37 |
0: [2023-03-15 23:19:12,427] [INFO] [logging.py:68:log_dist] [Rank 0] step=10000, skipped=0, lr=[2.568592118488235e-05, 2.568592118488235e-05, 2.568592118488235e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    10000/   11269 | consumed samples:      2560000 | consumed tokens:   5242880000 | elapsed time per iteration (s): 0.47 | learning rate: 2.569E-05 | global batch size:   256 | lm loss: 3.307291E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.728 | TFLOPs: 35.36 |
0: steps: 10000 loss: 3.3268 iter time (s): 0.475 samples/sec: 538.820
7: ------------------------------------------------------------------------------------------------
7: validation loss at iteration 10000 | lm loss value: 3.340298E+00 | lm loss PPL: 2.822753E+01 | 
7: ------------------------------------------------------------------------------------------------
0: saving checkpoint at iteration   10000 to checkpoints_280m5b9400m
0: [2023-03-15 23:19:12,605] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step10000 is begin to save!
0: [2023-03-15 23:19:12,609] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_01-model_00-model_states.pt...
0: [2023-03-15 23:19:12,728] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_01-model_00-model_states.pt.
0: [2023-03-15 23:19:12,729] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_03-model_00-model_states.pt...
0: [2023-03-15 23:19:12,754] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_03-model_00-model_states.pt.
0: [2023-03-15 23:19:12,754] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_04-model_00-model_states.pt...
0: [2023-03-15 23:19:12,779] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_04-model_00-model_states.pt.
0: [2023-03-15 23:19:12,779] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_05-model_00-model_states.pt...
0: [2023-03-15 23:19:12,803] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_05-model_00-model_states.pt.
0: [2023-03-15 23:19:12,804] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_06-model_00-model_states.pt...
0: [2023-03-15 23:19:12,828] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_06-model_00-model_states.pt.
0: [2023-03-15 23:19:12,828] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_07-model_00-model_states.pt...
0: [2023-03-15 23:19:12,853] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_07-model_00-model_states.pt.
0: [2023-03-15 23:19:12,853] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_08-model_00-model_states.pt...
0: [2023-03-15 23:19:12,877] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_08-model_00-model_states.pt.
0: [2023-03-15 23:19:12,878] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_09-model_00-model_states.pt...
0: [2023-03-15 23:19:12,904] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_09-model_00-model_states.pt.
0: [2023-03-15 23:19:12,904] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_10-model_00-model_states.pt...
0: [2023-03-15 23:19:12,929] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_10-model_00-model_states.pt.
0: [2023-03-15 23:19:12,929] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_11-model_00-model_states.pt...
0: [2023-03-15 23:19:12,953] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_11-model_00-model_states.pt.
0: [2023-03-15 23:19:12,954] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_12-model_00-model_states.pt...
0: [2023-03-15 23:19:12,978] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_12-model_00-model_states.pt.
0: [2023-03-15 23:19:12,978] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_13-model_00-model_states.pt...
0: [2023-03-15 23:19:13,003] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_13-model_00-model_states.pt.
0: [2023-03-15 23:19:13,003] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_14-model_00-model_states.pt...
0: [2023-03-15 23:19:13,027] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_14-model_00-model_states.pt.
0: [2023-03-15 23:19:13,027] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_15-model_00-model_states.pt...
0: [2023-03-15 23:19:13,052] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_15-model_00-model_states.pt.
0: [2023-03-15 23:19:13,052] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_16-model_00-model_states.pt...
0: [2023-03-15 23:19:13,076] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_16-model_00-model_states.pt.
0: [2023-03-15 23:19:13,077] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_17-model_00-model_states.pt...
0: [2023-03-15 23:19:13,101] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_17-model_00-model_states.pt.
0: [2023-03-15 23:19:13,101] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_18-model_00-model_states.pt...
0: [2023-03-15 23:19:13,126] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_18-model_00-model_states.pt.
0: [2023-03-15 23:19:13,126] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_19-model_00-model_states.pt...
0: [2023-03-15 23:19:13,150] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_19-model_00-model_states.pt.
0: [2023-03-15 23:19:13,150] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_20-model_00-model_states.pt...
0: [2023-03-15 23:19:13,175] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_20-model_00-model_states.pt.
0: [2023-03-15 23:19:13,175] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/layer_22-model_00-model_states.pt...
0: [2023-03-15 23:19:13,176] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/layer_22-model_00-model_states.pt.
0: [2023-03-15 23:19:13,177] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_280m5b9400m/global_step10000/mp_rank_00_model_states.pt
0: [2023-03-15 23:19:13,177] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/mp_rank_00_model_states.pt...
0: [2023-03-15 23:19:13,182] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/mp_rank_00_model_states.pt.
0: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:19:13,199] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:19:13,258] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:19:13,258] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-15 23:19:13,258] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
2: [2023-03-15 23:19:13,259] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:19:13,260] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-15 23:19:13,260] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-15 23:19:13,260] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:19:13,260] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-15 23:19:13,261] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
7: [2023-03-15 23:19:13,261] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:19:13,261] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-15 23:19:13,261] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
3: [2023-03-15 23:19:13,261] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:19:13,261] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-15 23:19:13,261] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-15 23:19:13,262] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:19:13,263] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-15 23:19:13,263] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-15 23:19:13,263] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:19:13,263] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-15 23:19:13,263] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
6: [2023-03-15 23:19:13,263] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:19:13,264] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:19:13,263] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:19:13,264] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-15 23:19:13,264] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
3: [2023-03-15 23:19:13,264] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
6: [2023-03-15 23:19:13,264] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-15 23:19:13,264] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:19:13,264] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:19:13,264] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
6: [2023-03-15 23:19:13,264] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
6: [2023-03-15 23:19:13,264] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-15 23:19:13,264] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-15 23:19:13,264] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
6: [2023-03-15 23:19:13,264] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
3: [2023-03-15 23:19:13,264] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:19:13,264] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
2: [2023-03-15 23:19:13,264] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:19:13,264] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-15 23:19:13,264] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:19:13,264] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-15 23:19:13,264] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-15 23:19:13,264] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-15 23:19:13,264] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
7: [2023-03-15 23:19:13,264] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:19:13,264] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-15 23:19:13,264] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
6: [2023-03-15 23:19:13,265] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:19:13,265] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-15 23:19:13,265] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
5: [2023-03-15 23:19:13,266] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:19:13,266] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-15 23:19:13,266] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-15 23:19:13,266] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:19:13,266] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:19:13,266] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-15 23:19:13,266] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:19:13,266] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
1: [2023-03-15 23:19:13,266] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-15 23:19:13,266] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
2: [2023-03-15 23:19:13,266] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-15 23:19:13,266] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-15 23:19:13,266] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:19:13,266] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-15 23:19:13,266] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
3: [2023-03-15 23:19:13,267] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:19:13,267] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-15 23:19:13,267] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-15 23:19:13,267] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:19:13,267] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-15 23:19:13,267] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
5: [2023-03-15 23:19:13,268] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:19:13,268] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-15 23:19:13,268] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
6: [2023-03-15 23:19:13,268] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:19:13,268] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-15 23:19:13,268] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
2: [2023-03-15 23:19:13,268] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:19:13,269] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-15 23:19:13,269] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
0: [2023-03-15 23:19:13,268] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:19:13,269] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-15 23:19:13,269] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
0: [2023-03-15 23:19:13,269] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:19:13,269] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
7: [2023-03-15 23:19:13,269] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:19:13,269] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:19:13,269] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:19:13,269] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:19:13,269] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-15 23:19:13,269] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-15 23:19:13,269] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-15 23:19:13,269] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
0: [2023-03-15 23:19:13,269] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-15 23:19:13,269] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:19:13,269] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
7: [2023-03-15 23:19:13,269] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
7: [2023-03-15 23:19:13,269] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
7: [2023-03-15 23:19:13,269] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-15 23:19:13,269] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-15 23:19:13,269] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
0: [2023-03-15 23:19:13,269] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:19:13,269] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-15 23:19:13,269] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
0: [2023-03-15 23:19:13,270] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:19:13,270] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-15 23:19:13,270] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
6: [2023-03-15 23:19:13,270] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:19:13,270] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-15 23:19:13,270] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
2: [2023-03-15 23:19:13,270] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:19:13,271] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-15 23:19:13,271] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
5: [2023-03-15 23:19:13,271] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:19:13,271] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-15 23:19:13,271] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
2: [2023-03-15 23:19:13,271] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:19:13,271] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-15 23:19:13,271] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
2: [2023-03-15 23:19:13,271] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:19:13,271] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-15 23:19:13,271] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
5: [2023-03-15 23:19:13,272] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:19:13,272] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:19:13,272] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-15 23:19:13,272] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-15 23:19:13,272] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
5: [2023-03-15 23:19:13,272] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
6: [2023-03-15 23:19:13,272] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:19:13,272] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-15 23:19:13,273] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
5: [2023-03-15 23:19:13,273] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:19:13,274] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-15 23:19:13,274] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
2: [2023-03-15 23:19:13,274] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:19:13,274] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-15 23:19:13,274] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-15 23:19:13,275] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:19:13,275] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-15 23:19:13,275] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-15 23:19:13,268] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:19:13,268] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-15 23:19:13,268] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-15 23:19:13,272] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:19:13,272] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-15 23:19:13,272] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-15 23:19:13,275] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:19:13,275] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-15 23:19:13,275] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
5: [2023-03-15 23:19:13,276] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:19:13,276] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:19:13,276] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-15 23:19:13,276] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-15 23:19:13,276] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
5: [2023-03-15 23:19:13,276] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
0: [2023-03-15 23:19:13,276] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:19:13,277] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:19:13,277] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-15 23:19:13,277] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
0: [2023-03-15 23:19:13,277] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:19:13,277] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-15 23:19:13,277] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
0: [2023-03-15 23:19:13,278] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:19:13,278] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-15 23:19:13,278] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
7: [2023-03-15 23:19:13,281] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:19:13,281] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-15 23:19:13,282] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
7: [2023-03-15 23:19:13,282] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:19:13,282] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-15 23:19:13,282] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-15 23:19:13,282] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:19:13,282] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-15 23:19:13,282] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-15 23:19:13,282] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:19:13,282] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-15 23:19:13,282] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
3: [2023-03-15 23:19:13,290] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:19:13,290] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-15 23:19:13,290] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
3: [2023-03-15 23:19:13,290] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:19:13,290] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-15 23:19:13,290] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
3: [2023-03-15 23:19:13,301] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:19:13,302] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-15 23:19:13,302] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
3: [2023-03-15 23:19:13,302] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:19:13,302] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-15 23:19:13,302] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
0: [2023-03-15 23:19:13,351] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step10000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-15 23:19:13,351] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
0:   successfully saved checkpoint at iteration   10000 to checkpoints_280m5b9400m
7: time (ms) | save-checkpoint: 751.66
7:  iteration    10010/   11269 | consumed samples:      2562560 | consumed tokens:   5248122880 | elapsed time per iteration (s): 0.57 | learning rate: 2.560E-05 | global batch size:   256 | lm loss: 3.310117E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 451.045 | TFLOPs: 29.50 |
7:  iteration    10020/   11269 | consumed samples:      2565120 | consumed tokens:   5253365760 | elapsed time per iteration (s): 0.47 | learning rate: 2.551E-05 | global batch size:   256 | lm loss: 3.308623E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.977 | TFLOPs: 35.45 |
7:  iteration    10030/   11269 | consumed samples:      2567680 | consumed tokens:   5258608640 | elapsed time per iteration (s): 0.47 | learning rate: 2.542E-05 | global batch size:   256 | lm loss: 3.313069E+00 | grad norm: 0.283 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.703 | TFLOPs: 35.43 |
7:  iteration    10040/   11269 | consumed samples:      2570240 | consumed tokens:   5263851520 | elapsed time per iteration (s): 0.47 | learning rate: 2.534E-05 | global batch size:   256 | lm loss: 3.312185E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.599 | TFLOPs: 35.42 |
7:  iteration    10050/   11269 | consumed samples:      2572800 | consumed tokens:   5269094400 | elapsed time per iteration (s): 0.47 | learning rate: 2.525E-05 | global batch size:   256 | lm loss: 3.316109E+00 | grad norm: 0.285 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.373 | TFLOPs: 35.41 |
7:  iteration    10060/   11269 | consumed samples:      2575360 | consumed tokens:   5274337280 | elapsed time per iteration (s): 0.47 | learning rate: 2.517E-05 | global batch size:   256 | lm loss: 3.312875E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.582 | TFLOPs: 35.42 |
7:  iteration    10070/   11269 | consumed samples:      2577920 | consumed tokens:   5279580160 | elapsed time per iteration (s): 0.47 | learning rate: 2.508E-05 | global batch size:   256 | lm loss: 3.325964E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.485 | TFLOPs: 35.41 |
7:  iteration    10080/   11269 | consumed samples:      2580480 | consumed tokens:   5284823040 | elapsed time per iteration (s): 0.47 | learning rate: 2.500E-05 | global batch size:   256 | lm loss: 3.313785E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.009 | TFLOPs: 35.38 |
7:  iteration    10090/   11269 | consumed samples:      2583040 | consumed tokens:   5290065920 | elapsed time per iteration (s): 0.47 | learning rate: 2.492E-05 | global batch size:   256 | lm loss: 3.325539E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.296 | TFLOPs: 35.40 |
7:  iteration    10100/   11269 | consumed samples:      2585600 | consumed tokens:   5295308800 | elapsed time per iteration (s): 0.48 | learning rate: 2.483E-05 | global batch size:   256 | lm loss: 3.312859E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.861 | TFLOPs: 35.11 |
7:  iteration    10110/   11269 | consumed samples:      2588160 | consumed tokens:   5300551680 | elapsed time per iteration (s): 0.47 | learning rate: 2.475E-05 | global batch size:   256 | lm loss: 3.296546E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.080 | TFLOPs: 35.39 |
7:  iteration    10120/   11269 | consumed samples:      2590720 | consumed tokens:   5305794560 | elapsed time per iteration (s): 0.47 | learning rate: 2.467E-05 | global batch size:   256 | lm loss: 3.320692E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.816 | TFLOPs: 35.37 |
7:  iteration    10130/   11269 | consumed samples:      2593280 | consumed tokens:   5311037440 | elapsed time per iteration (s): 0.47 | learning rate: 2.459E-05 | global batch size:   256 | lm loss: 3.289388E+00 | grad norm: 0.285 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.113 | TFLOPs: 35.39 |
7:  iteration    10140/   11269 | consumed samples:      2595840 | consumed tokens:   5316280320 | elapsed time per iteration (s): 0.47 | learning rate: 2.451E-05 | global batch size:   256 | lm loss: 3.302771E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.161 | TFLOPs: 35.39 |
7:  iteration    10150/   11269 | consumed samples:      2598400 | consumed tokens:   5321523200 | elapsed time per iteration (s): 0.47 | learning rate: 2.443E-05 | global batch size:   256 | lm loss: 3.295461E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.790 | TFLOPs: 35.37 |
7:  iteration    10160/   11269 | consumed samples:      2600960 | consumed tokens:   5326766080 | elapsed time per iteration (s): 0.47 | learning rate: 2.435E-05 | global batch size:   256 | lm loss: 3.311135E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.957 | TFLOPs: 35.38 |
7:  iteration    10170/   11269 | consumed samples:      2603520 | consumed tokens:   5332008960 | elapsed time per iteration (s): 0.47 | learning rate: 2.428E-05 | global batch size:   256 | lm loss: 3.303082E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.908 | TFLOPs: 35.38 |
7:  iteration    10180/   11269 | consumed samples:      2606080 | consumed tokens:   5337251840 | elapsed time per iteration (s): 0.47 | learning rate: 2.420E-05 | global batch size:   256 | lm loss: 3.313264E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.111 | TFLOPs: 35.39 |
7:  iteration    10190/   11269 | consumed samples:      2608640 | consumed tokens:   5342494720 | elapsed time per iteration (s): 0.47 | learning rate: 2.412E-05 | global batch size:   256 | lm loss: 3.295168E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.840 | TFLOPs: 35.37 |
7:  iteration    10200/   11269 | consumed samples:      2611200 | consumed tokens:   5347737600 | elapsed time per iteration (s): 0.47 | learning rate: 2.405E-05 | global batch size:   256 | lm loss: 3.304145E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.866 | TFLOPs: 35.37 |
7:  iteration    10210/   11269 | consumed samples:      2613760 | consumed tokens:   5352980480 | elapsed time per iteration (s): 0.47 | learning rate: 2.397E-05 | global batch size:   256 | lm loss: 3.318631E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.068 | TFLOPs: 35.39 |
7:  iteration    10220/   11269 | consumed samples:      2616320 | consumed tokens:   5358223360 | elapsed time per iteration (s): 0.47 | learning rate: 2.390E-05 | global batch size:   256 | lm loss: 3.312860E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.738 | TFLOPs: 35.36 |
7:  iteration    10230/   11269 | consumed samples:      2618880 | consumed tokens:   5363466240 | elapsed time per iteration (s): 0.47 | learning rate: 2.383E-05 | global batch size:   256 | lm loss: 3.295499E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.909 | TFLOPs: 35.38 |
7:  iteration    10240/   11269 | consumed samples:      2621440 | consumed tokens:   5368709120 | elapsed time per iteration (s): 0.47 | learning rate: 2.375E-05 | global batch size:   256 | lm loss: 3.308125E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.851 | TFLOPs: 35.37 |
7:  iteration    10250/   11269 | consumed samples:      2624000 | consumed tokens:   5373952000 | elapsed time per iteration (s): 0.47 | learning rate: 2.368E-05 | global batch size:   256 | lm loss: 3.305819E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.688 | TFLOPs: 35.36 |
7:  iteration    10260/   11269 | consumed samples:      2626560 | consumed tokens:   5379194880 | elapsed time per iteration (s): 0.47 | learning rate: 2.361E-05 | global batch size:   256 | lm loss: 3.300026E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.991 | TFLOPs: 35.32 |
7:  iteration    10270/   11269 | consumed samples:      2629120 | consumed tokens:   5384437760 | elapsed time per iteration (s): 0.48 | learning rate: 2.354E-05 | global batch size:   256 | lm loss: 3.294808E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 537.516 | TFLOPs: 35.15 |
7:  iteration    10280/   11269 | consumed samples:      2631680 | consumed tokens:   5389680640 | elapsed time per iteration (s): 0.47 | learning rate: 2.347E-05 | global batch size:   256 | lm loss: 3.304939E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.777 | TFLOPs: 35.37 |
7:  iteration    10290/   11269 | consumed samples:      2634240 | consumed tokens:   5394923520 | elapsed time per iteration (s): 0.47 | learning rate: 2.340E-05 | global batch size:   256 | lm loss: 3.295316E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.950 | TFLOPs: 35.38 |
7:  iteration    10300/   11269 | consumed samples:      2636800 | consumed tokens:   5400166400 | elapsed time per iteration (s): 0.47 | learning rate: 2.333E-05 | global batch size:   256 | lm loss: 3.313411E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.730 | TFLOPs: 35.36 |
7:  iteration    10310/   11269 | consumed samples:      2639360 | consumed tokens:   5405409280 | elapsed time per iteration (s): 0.47 | learning rate: 2.326E-05 | global batch size:   256 | lm loss: 3.303754E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.174 | TFLOPs: 35.39 |
7:  iteration    10320/   11269 | consumed samples:      2641920 | consumed tokens:   5410652160 | elapsed time per iteration (s): 0.47 | learning rate: 2.319E-05 | global batch size:   256 | lm loss: 3.307568E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.560 | TFLOPs: 35.35 |
7:  iteration    10330/   11269 | consumed samples:      2644480 | consumed tokens:   5415895040 | elapsed time per iteration (s): 0.47 | learning rate: 2.313E-05 | global batch size:   256 | lm loss: 3.315317E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.862 | TFLOPs: 35.37 |
7:  iteration    10340/   11269 | consumed samples:      2647040 | consumed tokens:   5421137920 | elapsed time per iteration (s): 0.47 | learning rate: 2.306E-05 | global batch size:   256 | lm loss: 3.295657E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.915 | TFLOPs: 35.38 |
7:  iteration    10350/   11269 | consumed samples:      2649600 | consumed tokens:   5426380800 | elapsed time per iteration (s): 0.47 | learning rate: 2.300E-05 | global batch size:   256 | lm loss: 3.286523E+00 | grad norm: 0.285 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.042 | TFLOPs: 35.38 |
7:  iteration    10360/   11269 | consumed samples:      2652160 | consumed tokens:   5431623680 | elapsed time per iteration (s): 0.47 | learning rate: 2.293E-05 | global batch size:   256 | lm loss: 3.302799E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.518 | TFLOPs: 35.35 |
7:  iteration    10370/   11269 | consumed samples:      2654720 | consumed tokens:   5436866560 | elapsed time per iteration (s): 0.47 | learning rate: 2.287E-05 | global batch size:   256 | lm loss: 3.309127E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.785 | TFLOPs: 35.37 |
7:  iteration    10380/   11269 | consumed samples:      2657280 | consumed tokens:   5442109440 | elapsed time per iteration (s): 0.47 | learning rate: 2.281E-05 | global batch size:   256 | lm loss: 3.296850E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.727 | TFLOPs: 35.36 |
7:  iteration    10390/   11269 | consumed samples:      2659840 | consumed tokens:   5447352320 | elapsed time per iteration (s): 0.47 | learning rate: 2.274E-05 | global batch size:   256 | lm loss: 3.283238E+00 | grad norm: 0.284 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.796 | TFLOPs: 35.37 |
7:  iteration    10400/   11269 | consumed samples:      2662400 | consumed tokens:   5452595200 | elapsed time per iteration (s): 0.47 | learning rate: 2.268E-05 | global batch size:   256 | lm loss: 3.308766E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.795 | TFLOPs: 35.37 |
7:  iteration    10410/   11269 | consumed samples:      2664960 | consumed tokens:   5457838080 | elapsed time per iteration (s): 0.47 | learning rate: 2.262E-05 | global batch size:   256 | lm loss: 3.303588E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.498 | TFLOPs: 35.35 |
7:  iteration    10420/   11269 | consumed samples:      2667520 | consumed tokens:   5463080960 | elapsed time per iteration (s): 0.47 | learning rate: 2.256E-05 | global batch size:   256 | lm loss: 3.298882E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.800 | TFLOPs: 35.37 |
7:  iteration    10430/   11269 | consumed samples:      2670080 | consumed tokens:   5468323840 | elapsed time per iteration (s): 0.47 | learning rate: 2.250E-05 | global batch size:   256 | lm loss: 3.303219E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.172 | TFLOPs: 35.39 |
7:  iteration    10440/   11269 | consumed samples:      2672640 | consumed tokens:   5473566720 | elapsed time per iteration (s): 0.47 | learning rate: 2.244E-05 | global batch size:   256 | lm loss: 3.306059E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.618 | TFLOPs: 35.36 |
7:  iteration    10450/   11269 | consumed samples:      2675200 | consumed tokens:   5478809600 | elapsed time per iteration (s): 0.47 | learning rate: 2.238E-05 | global batch size:   256 | lm loss: 3.300653E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.954 | TFLOPs: 35.38 |
7:  iteration    10460/   11269 | consumed samples:      2677760 | consumed tokens:   5484052480 | elapsed time per iteration (s): 0.47 | learning rate: 2.233E-05 | global batch size:   256 | lm loss: 3.296422E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.595 | TFLOPs: 35.35 |
7:  iteration    10470/   11269 | consumed samples:      2680320 | consumed tokens:   5489295360 | elapsed time per iteration (s): 0.47 | learning rate: 2.227E-05 | global batch size:   256 | lm loss: 3.305302E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.604 | TFLOPs: 35.36 |
7:  iteration    10480/   11269 | consumed samples:      2682880 | consumed tokens:   5494538240 | elapsed time per iteration (s): 0.47 | learning rate: 2.221E-05 | global batch size:   256 | lm loss: 3.297302E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.583 | TFLOPs: 35.35 |
7:  iteration    10490/   11269 | consumed samples:      2685440 | consumed tokens:   5499781120 | elapsed time per iteration (s): 0.47 | learning rate: 2.216E-05 | global batch size:   256 | lm loss: 3.300962E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.658 | TFLOPs: 35.36 |
7:  iteration    10500/   11269 | consumed samples:      2688000 | consumed tokens:   5505024000 | elapsed time per iteration (s): 0.47 | learning rate: 2.210E-05 | global batch size:   256 | lm loss: 3.302254E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.729 | TFLOPs: 35.36 |
7:  iteration    10510/   11269 | consumed samples:      2690560 | consumed tokens:   5510266880 | elapsed time per iteration (s): 0.47 | learning rate: 2.205E-05 | global batch size:   256 | lm loss: 3.291319E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.934 | TFLOPs: 35.38 |
7:  iteration    10520/   11269 | consumed samples:      2693120 | consumed tokens:   5515509760 | elapsed time per iteration (s): 0.47 | learning rate: 2.199E-05 | global batch size:   256 | lm loss: 3.289344E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.003 | TFLOPs: 35.38 |
7:  iteration    10530/   11269 | consumed samples:      2695680 | consumed tokens:   5520752640 | elapsed time per iteration (s): 0.47 | learning rate: 2.194E-05 | global batch size:   256 | lm loss: 3.306117E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.999 | TFLOPs: 35.38 |
7:  iteration    10540/   11269 | consumed samples:      2698240 | consumed tokens:   5525995520 | elapsed time per iteration (s): 0.47 | learning rate: 2.189E-05 | global batch size:   256 | lm loss: 3.315158E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.766 | TFLOPs: 35.37 |
7:  iteration    10550/   11269 | consumed samples:      2700800 | consumed tokens:   5531238400 | elapsed time per iteration (s): 0.47 | learning rate: 2.184E-05 | global batch size:   256 | lm loss: 3.286473E+00 | grad norm: 0.286 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.952 | TFLOPs: 35.38 |
7:  iteration    10560/   11269 | consumed samples:      2703360 | consumed tokens:   5536481280 | elapsed time per iteration (s): 0.47 | learning rate: 2.179E-05 | global batch size:   256 | lm loss: 3.297543E+00 | grad norm: 0.285 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.377 | TFLOPs: 35.34 |
7:  iteration    10570/   11269 | consumed samples:      2705920 | consumed tokens:   5541724160 | elapsed time per iteration (s): 0.47 | learning rate: 2.174E-05 | global batch size:   256 | lm loss: 3.311323E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.616 | TFLOPs: 35.36 |
7:  iteration    10580/   11269 | consumed samples:      2708480 | consumed tokens:   5546967040 | elapsed time per iteration (s): 0.47 | learning rate: 2.169E-05 | global batch size:   256 | lm loss: 3.293797E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.848 | TFLOPs: 35.37 |
7:  iteration    10590/   11269 | consumed samples:      2711040 | consumed tokens:   5552209920 | elapsed time per iteration (s): 0.47 | learning rate: 2.164E-05 | global batch size:   256 | lm loss: 3.308405E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.585 | TFLOPs: 35.35 |
7:  iteration    10600/   11269 | consumed samples:      2713600 | consumed tokens:   5557452800 | elapsed time per iteration (s): 0.47 | learning rate: 2.159E-05 | global batch size:   256 | lm loss: 3.293490E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.964 | TFLOPs: 35.38 |
7:  iteration    10610/   11269 | consumed samples:      2716160 | consumed tokens:   5562695680 | elapsed time per iteration (s): 0.47 | learning rate: 2.155E-05 | global batch size:   256 | lm loss: 3.298980E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.970 | TFLOPs: 35.38 |
7:  iteration    10620/   11269 | consumed samples:      2718720 | consumed tokens:   5567938560 | elapsed time per iteration (s): 0.47 | learning rate: 2.150E-05 | global batch size:   256 | lm loss: 3.305222E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.959 | TFLOPs: 35.38 |
7:  iteration    10630/   11269 | consumed samples:      2721280 | consumed tokens:   5573181440 | elapsed time per iteration (s): 0.47 | learning rate: 2.145E-05 | global batch size:   256 | lm loss: 3.303561E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.665 | TFLOPs: 35.36 |
7:  iteration    10640/   11269 | consumed samples:      2723840 | consumed tokens:   5578424320 | elapsed time per iteration (s): 0.47 | learning rate: 2.141E-05 | global batch size:   256 | lm loss: 3.306315E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.537 | TFLOPs: 35.35 |
7:  iteration    10650/   11269 | consumed samples:      2726400 | consumed tokens:   5583667200 | elapsed time per iteration (s): 0.47 | learning rate: 2.136E-05 | global batch size:   256 | lm loss: 3.311818E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.724 | TFLOPs: 35.36 |
7:  iteration    10660/   11269 | consumed samples:      2728960 | consumed tokens:   5588910080 | elapsed time per iteration (s): 0.47 | learning rate: 2.132E-05 | global batch size:   256 | lm loss: 3.306777E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.417 | TFLOPs: 35.41 |
7:  iteration    10670/   11269 | consumed samples:      2731520 | consumed tokens:   5594152960 | elapsed time per iteration (s): 0.47 | learning rate: 2.128E-05 | global batch size:   256 | lm loss: 3.292141E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.352 | TFLOPs: 35.34 |
7:  iteration    10680/   11269 | consumed samples:      2734080 | consumed tokens:   5599395840 | elapsed time per iteration (s): 0.48 | learning rate: 2.124E-05 | global batch size:   256 | lm loss: 3.285120E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 536.454 | TFLOPs: 35.08 |
7:  iteration    10690/   11269 | consumed samples:      2736640 | consumed tokens:   5604638720 | elapsed time per iteration (s): 0.47 | learning rate: 2.119E-05 | global batch size:   256 | lm loss: 3.290388E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.660 | TFLOPs: 35.36 |
7:  iteration    10700/   11269 | consumed samples:      2739200 | consumed tokens:   5609881600 | elapsed time per iteration (s): 0.47 | learning rate: 2.115E-05 | global batch size:   256 | lm loss: 3.279569E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.553 | TFLOPs: 35.35 |
7:  iteration    10710/   11269 | consumed samples:      2741760 | consumed tokens:   5615124480 | elapsed time per iteration (s): 0.47 | learning rate: 2.111E-05 | global batch size:   256 | lm loss: 3.295605E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.507 | TFLOPs: 35.35 |
7:  iteration    10720/   11269 | consumed samples:      2744320 | consumed tokens:   5620367360 | elapsed time per iteration (s): 0.47 | learning rate: 2.107E-05 | global batch size:   256 | lm loss: 3.292328E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.655 | TFLOPs: 35.36 |
7:  iteration    10730/   11269 | consumed samples:      2746880 | consumed tokens:   5625610240 | elapsed time per iteration (s): 0.48 | learning rate: 2.103E-05 | global batch size:   256 | lm loss: 3.289630E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.159 | TFLOPs: 35.20 |
7:  iteration    10740/   11269 | consumed samples:      2749440 | consumed tokens:   5630853120 | elapsed time per iteration (s): 0.47 | learning rate: 2.100E-05 | global batch size:   256 | lm loss: 3.291000E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.674 | TFLOPs: 35.36 |
7:  iteration    10750/   11269 | consumed samples:      2752000 | consumed tokens:   5636096000 | elapsed time per iteration (s): 0.47 | learning rate: 2.096E-05 | global batch size:   256 | lm loss: 3.287538E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.613 | TFLOPs: 35.36 |
7:  iteration    10760/   11269 | consumed samples:      2754560 | consumed tokens:   5641338880 | elapsed time per iteration (s): 0.47 | learning rate: 2.092E-05 | global batch size:   256 | lm loss: 3.280148E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.394 | TFLOPs: 35.34 |
7:  iteration    10770/   11269 | consumed samples:      2757120 | consumed tokens:   5646581760 | elapsed time per iteration (s): 0.47 | learning rate: 2.089E-05 | global batch size:   256 | lm loss: 3.293938E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.559 | TFLOPs: 35.35 |
7:  iteration    10780/   11269 | consumed samples:      2759680 | consumed tokens:   5651824640 | elapsed time per iteration (s): 0.47 | learning rate: 2.085E-05 | global batch size:   256 | lm loss: 3.286836E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.629 | TFLOPs: 35.36 |
7:  iteration    10790/   11269 | consumed samples:      2762240 | consumed tokens:   5657067520 | elapsed time per iteration (s): 0.47 | learning rate: 2.082E-05 | global batch size:   256 | lm loss: 3.289599E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.556 | TFLOPs: 35.35 |
7:  iteration    10800/   11269 | consumed samples:      2764800 | consumed tokens:   5662310400 | elapsed time per iteration (s): 0.47 | learning rate: 2.078E-05 | global batch size:   256 | lm loss: 3.293338E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.885 | TFLOPs: 35.37 |
7:  iteration    10810/   11269 | consumed samples:      2767360 | consumed tokens:   5667553280 | elapsed time per iteration (s): 0.47 | learning rate: 2.075E-05 | global batch size:   256 | lm loss: 3.289097E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.139 | TFLOPs: 35.39 |
7:  iteration    10820/   11269 | consumed samples:      2769920 | consumed tokens:   5672796160 | elapsed time per iteration (s): 0.47 | learning rate: 2.072E-05 | global batch size:   256 | lm loss: 3.302684E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.839 | TFLOPs: 35.37 |
7:  iteration    10830/   11269 | consumed samples:      2772480 | consumed tokens:   5678039040 | elapsed time per iteration (s): 0.47 | learning rate: 2.069E-05 | global batch size:   256 | lm loss: 3.291480E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.923 | TFLOPs: 35.38 |
7:  iteration    10840/   11269 | consumed samples:      2775040 | consumed tokens:   5683281920 | elapsed time per iteration (s): 0.47 | learning rate: 2.066E-05 | global batch size:   256 | lm loss: 3.288485E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.843 | TFLOPs: 35.37 |
7:  iteration    10850/   11269 | consumed samples:      2777600 | consumed tokens:   5688524800 | elapsed time per iteration (s): 0.47 | learning rate: 2.063E-05 | global batch size:   256 | lm loss: 3.274915E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.642 | TFLOPs: 35.36 |
7:  iteration    10860/   11269 | consumed samples:      2780160 | consumed tokens:   5693767680 | elapsed time per iteration (s): 0.47 | learning rate: 2.060E-05 | global batch size:   256 | lm loss: 3.298071E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.565 | TFLOPs: 35.35 |
7:  iteration    10870/   11269 | consumed samples:      2782720 | consumed tokens:   5699010560 | elapsed time per iteration (s): 0.47 | learning rate: 2.057E-05 | global batch size:   256 | lm loss: 3.281837E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.730 | TFLOPs: 35.36 |
7:  iteration    10880/   11269 | consumed samples:      2785280 | consumed tokens:   5704253440 | elapsed time per iteration (s): 0.47 | learning rate: 2.054E-05 | global batch size:   256 | lm loss: 3.288931E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.722 | TFLOPs: 35.36 |
7:  iteration    10890/   11269 | consumed samples:      2787840 | consumed tokens:   5709496320 | elapsed time per iteration (s): 0.47 | learning rate: 2.051E-05 | global batch size:   256 | lm loss: 3.294603E+00 | grad norm: 0.286 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.957 | TFLOPs: 35.38 |
7:  iteration    10900/   11269 | consumed samples:      2790400 | consumed tokens:   5714739200 | elapsed time per iteration (s): 0.47 | learning rate: 2.049E-05 | global batch size:   256 | lm loss: 3.293974E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.275 | TFLOPs: 35.33 |
7:  iteration    10910/   11269 | consumed samples:      2792960 | consumed tokens:   5719982080 | elapsed time per iteration (s): 0.47 | learning rate: 2.046E-05 | global batch size:   256 | lm loss: 3.290797E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.913 | TFLOPs: 35.38 |
7:  iteration    10920/   11269 | consumed samples:      2795520 | consumed tokens:   5725224960 | elapsed time per iteration (s): 0.47 | learning rate: 2.043E-05 | global batch size:   256 | lm loss: 3.275294E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.667 | TFLOPs: 35.36 |
7:  iteration    10930/   11269 | consumed samples:      2798080 | consumed tokens:   5730467840 | elapsed time per iteration (s): 0.47 | learning rate: 2.041E-05 | global batch size:   256 | lm loss: 3.280841E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.583 | TFLOPs: 35.35 |
7:  iteration    10940/   11269 | consumed samples:      2800640 | consumed tokens:   5735710720 | elapsed time per iteration (s): 0.47 | learning rate: 2.039E-05 | global batch size:   256 | lm loss: 3.302108E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.547 | TFLOPs: 35.35 |
7:  iteration    10950/   11269 | consumed samples:      2803200 | consumed tokens:   5740953600 | elapsed time per iteration (s): 0.47 | learning rate: 2.036E-05 | global batch size:   256 | lm loss: 3.282794E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.658 | TFLOPs: 35.36 |
7:  iteration    10960/   11269 | consumed samples:      2805760 | consumed tokens:   5746196480 | elapsed time per iteration (s): 0.47 | learning rate: 2.034E-05 | global batch size:   256 | lm loss: 3.298049E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.865 | TFLOPs: 35.37 |
7:  iteration    10970/   11269 | consumed samples:      2808320 | consumed tokens:   5751439360 | elapsed time per iteration (s): 0.47 | learning rate: 2.032E-05 | global batch size:   256 | lm loss: 3.299276E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.550 | TFLOPs: 35.35 |
7:  iteration    10980/   11269 | consumed samples:      2810880 | consumed tokens:   5756682240 | elapsed time per iteration (s): 0.47 | learning rate: 2.030E-05 | global batch size:   256 | lm loss: 3.288021E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.291 | TFLOPs: 35.33 |
7:  iteration    10990/   11269 | consumed samples:      2813440 | consumed tokens:   5761925120 | elapsed time per iteration (s): 0.47 | learning rate: 2.028E-05 | global batch size:   256 | lm loss: 3.284876E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.381 | TFLOPs: 35.34 |
7:  iteration    11000/   11269 | consumed samples:      2816000 | consumed tokens:   5767168000 | elapsed time per iteration (s): 0.47 | learning rate: 2.026E-05 | global batch size:   256 | lm loss: 3.287982E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.451 | TFLOPs: 35.35 |
7: ------------------------------------------------------------------------------------------------
7: validation loss at iteration 11000 | lm loss value: 3.441182E+00 | lm loss PPL: 3.122385E+01 | 
7: ------------------------------------------------------------------------------------------------
0: saving checkpoint at iteration   11000 to checkpoints_280m5b9400m
0: [2023-03-15 23:27:07,013] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step11000 is begin to save!
0: [2023-03-15 23:27:07,016] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_01-model_00-model_states.pt...
0: [2023-03-15 23:27:07,131] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_01-model_00-model_states.pt.
0: [2023-03-15 23:27:07,132] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_03-model_00-model_states.pt...
0: [2023-03-15 23:27:07,157] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_03-model_00-model_states.pt.
0: [2023-03-15 23:27:07,157] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_04-model_00-model_states.pt...
0: [2023-03-15 23:27:07,181] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_04-model_00-model_states.pt.
0: [2023-03-15 23:27:07,181] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_05-model_00-model_states.pt...
0: [2023-03-15 23:27:07,205] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_05-model_00-model_states.pt.
0: [2023-03-15 23:27:07,205] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_06-model_00-model_states.pt...
0: [2023-03-15 23:27:07,229] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_06-model_00-model_states.pt.
0: [2023-03-15 23:27:07,230] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_07-model_00-model_states.pt...
0: [2023-03-15 23:27:07,254] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_07-model_00-model_states.pt.
0: [2023-03-15 23:27:07,254] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_08-model_00-model_states.pt...
0: [2023-03-15 23:27:07,278] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_08-model_00-model_states.pt.
0: [2023-03-15 23:27:07,278] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_09-model_00-model_states.pt...
0: [2023-03-15 23:27:07,303] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_09-model_00-model_states.pt.
0: [2023-03-15 23:27:07,303] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_10-model_00-model_states.pt...
0: [2023-03-15 23:27:07,327] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_10-model_00-model_states.pt.
0: [2023-03-15 23:27:07,327] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_11-model_00-model_states.pt...
0: [2023-03-15 23:27:07,351] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_11-model_00-model_states.pt.
0: [2023-03-15 23:27:07,352] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_12-model_00-model_states.pt...
0: [2023-03-15 23:27:07,376] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_12-model_00-model_states.pt.
0: [2023-03-15 23:27:07,376] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_13-model_00-model_states.pt...
0: [2023-03-15 23:27:07,402] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_13-model_00-model_states.pt.
0: [2023-03-15 23:27:07,402] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_14-model_00-model_states.pt...
0: [2023-03-15 23:27:07,426] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_14-model_00-model_states.pt.
0: [2023-03-15 23:27:07,427] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_15-model_00-model_states.pt...
0: [2023-03-15 23:27:07,451] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_15-model_00-model_states.pt.
0: [2023-03-15 23:27:07,451] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_16-model_00-model_states.pt...
0: [2023-03-15 23:27:07,475] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_16-model_00-model_states.pt.
0: [2023-03-15 23:27:07,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_17-model_00-model_states.pt...
0: [2023-03-15 23:27:07,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_17-model_00-model_states.pt.
0: [2023-03-15 23:27:07,500] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_18-model_00-model_states.pt...
0: [2023-03-15 23:27:07,524] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_18-model_00-model_states.pt.
0: [2023-03-15 23:27:07,524] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_19-model_00-model_states.pt...
0: [2023-03-15 23:27:07,549] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_19-model_00-model_states.pt.
0: [2023-03-15 23:27:07,549] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_20-model_00-model_states.pt...
0: [2023-03-15 23:27:07,573] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_20-model_00-model_states.pt.
0: [2023-03-15 23:27:07,573] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/layer_22-model_00-model_states.pt...
0: [2023-03-15 23:27:07,574] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/layer_22-model_00-model_states.pt.
0: [2023-03-15 23:27:07,575] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_280m5b9400m/global_step11000/mp_rank_00_model_states.pt
0: [2023-03-15 23:27:07,575] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/mp_rank_00_model_states.pt...
0: [2023-03-15 23:27:07,578] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/mp_rank_00_model_states.pt.
0: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:27:07,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:27:07,654] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:27:07,654] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:27:07,654] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:27:07,654] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
5: [2023-03-15 23:27:07,654] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
2: [2023-03-15 23:27:07,654] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
5: [2023-03-15 23:27:07,654] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
7: [2023-03-15 23:27:07,655] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:27:07,655] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-15 23:27:07,655] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
5: [2023-03-15 23:27:07,657] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:27:07,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-15 23:27:07,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
6: [2023-03-15 23:27:07,657] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:27:07,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-15 23:27:07,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
5: [2023-03-15 23:27:07,657] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:27:07,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-15 23:27:07,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
3: [2023-03-15 23:27:07,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:27:07,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:27:07,659] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-15 23:27:07,659] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
6: [2023-03-15 23:27:07,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:27:07,659] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-15 23:27:07,659] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
1: [2023-03-15 23:27:07,660] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:27:07,660] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-15 23:27:07,661] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
5: [2023-03-15 23:27:07,661] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:27:07,661] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-15 23:27:07,661] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
7: [2023-03-15 23:27:07,661] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:27:07,661] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-15 23:27:07,661] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
1: [2023-03-15 23:27:07,662] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:27:07,662] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-15 23:27:07,662] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:27:07,662] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
1: [2023-03-15 23:27:07,662] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-15 23:27:07,662] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
6: [2023-03-15 23:27:07,662] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:27:07,662] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-15 23:27:07,662] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
6: [2023-03-15 23:27:07,662] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:27:07,663] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
5: [2023-03-15 23:27:07,663] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:27:07,663] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
5: [2023-03-15 23:27:07,663] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-15 23:27:07,663] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
0: [2023-03-15 23:27:07,664] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:27:07,664] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-15 23:27:07,664] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
0: [2023-03-15 23:27:07,664] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:27:07,664] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-15 23:27:07,664] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
0: [2023-03-15 23:27:07,664] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:27:07,664] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-15 23:27:07,664] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
0: [2023-03-15 23:27:07,666] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:27:07,666] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-15 23:27:07,666] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
6: [2023-03-15 23:27:07,666] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:27:07,666] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-15 23:27:07,666] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
6: [2023-03-15 23:27:07,667] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:27:07,667] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-15 23:27:07,667] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
1: [2023-03-15 23:27:07,668] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:27:07,668] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:27:07,668] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-15 23:27:07,668] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-15 23:27:07,668] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
1: [2023-03-15 23:27:07,668] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
6: [2023-03-15 23:27:07,668] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:27:07,668] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-15 23:27:07,668] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
5: [2023-03-15 23:27:07,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:27:07,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:27:07,669] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-15 23:27:07,669] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-15 23:27:07,669] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
5: [2023-03-15 23:27:07,669] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
1: [2023-03-15 23:27:07,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:27:07,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:27:07,669] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-15 23:27:07,669] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-15 23:27:07,669] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
1: [2023-03-15 23:27:07,669] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
7: [2023-03-15 23:27:07,670] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:27:07,670] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:27:07,670] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-15 23:27:07,670] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
0: [2023-03-15 23:27:07,670] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:27:07,670] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:27:07,670] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:27:07,670] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
7: [2023-03-15 23:27:07,670] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
0: [2023-03-15 23:27:07,671] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-15 23:27:07,671] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-15 23:27:07,671] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-15 23:27:07,671] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
0: [2023-03-15 23:27:07,671] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
0: [2023-03-15 23:27:07,671] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
5: [2023-03-15 23:27:07,673] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:27:07,673] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-15 23:27:07,673] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
7: [2023-03-15 23:27:07,675] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:27:07,675] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-15 23:27:07,675] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
6: [2023-03-15 23:27:07,675] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:27:07,675] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-15 23:27:07,675] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
4: [2023-03-15 23:27:07,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:27:07,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-15 23:27:07,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
3: [2023-03-15 23:27:07,661] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:27:07,661] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
4: [2023-03-15 23:27:07,660] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
3: [2023-03-15 23:27:07,661] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
4: [2023-03-15 23:27:07,660] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
3: [2023-03-15 23:27:07,661] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:27:07,660] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:27:07,661] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
4: [2023-03-15 23:27:07,660] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
3: [2023-03-15 23:27:07,661] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
4: [2023-03-15 23:27:07,660] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
3: [2023-03-15 23:27:07,671] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:27:07,662] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:27:07,671] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:27:07,663] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
3: [2023-03-15 23:27:07,671] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
4: [2023-03-15 23:27:07,663] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
3: [2023-03-15 23:27:07,671] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
4: [2023-03-15 23:27:07,664] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:27:07,671] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
4: [2023-03-15 23:27:07,664] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
3: [2023-03-15 23:27:07,671] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
4: [2023-03-15 23:27:07,665] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
4: [2023-03-15 23:27:07,666] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:27:07,666] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-15 23:27:07,666] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
4: [2023-03-15 23:27:07,667] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:27:07,667] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-15 23:27:07,667] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
4: [2023-03-15 23:27:07,673] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:27:07,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-15 23:27:07,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
4: [2023-03-15 23:27:07,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:27:07,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-15 23:27:07,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
2: [2023-03-15 23:27:07,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:27:07,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-15 23:27:07,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
2: [2023-03-15 23:27:07,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:27:07,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:27:07,669] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-15 23:27:07,669] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-15 23:27:07,669] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
2: [2023-03-15 23:27:07,669] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
2: [2023-03-15 23:27:07,673] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:27:07,673] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-15 23:27:07,673] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
2: [2023-03-15 23:27:07,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:27:07,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-15 23:27:07,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
2: [2023-03-15 23:27:07,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:27:07,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-15 23:27:07,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
2: [2023-03-15 23:27:07,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:27:07,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-15 23:27:07,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
3: [2023-03-15 23:27:07,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:27:07,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-15 23:27:07,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
3: [2023-03-15 23:27:07,675] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:27:07,675] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-15 23:27:07,675] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
3: [2023-03-15 23:27:07,676] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:27:07,676] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-15 23:27:07,676] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
0: [2023-03-15 23:27:07,692] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-15 23:27:07,692] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
7: [2023-03-15 23:27:07,714] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:27:07,714] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-15 23:27:07,714] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
7: [2023-03-15 23:27:07,714] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:27:07,714] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-15 23:27:07,714] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
7: [2023-03-15 23:27:07,715] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:27:07,715] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-15 23:27:07,715] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11000 is ready now!
0:   successfully saved checkpoint at iteration   11000 to checkpoints_280m5b9400m
7: time (ms) | save-checkpoint: 706.26
7:  iteration    11010/   11269 | consumed samples:      2818560 | consumed tokens:   5772410880 | elapsed time per iteration (s): 0.56 | learning rate: 2.024E-05 | global batch size:   256 | lm loss: 3.290878E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 457.014 | TFLOPs: 29.89 |
7:  iteration    11020/   11269 | consumed samples:      2821120 | consumed tokens:   5777653760 | elapsed time per iteration (s): 0.47 | learning rate: 2.022E-05 | global batch size:   256 | lm loss: 3.295529E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 542.055 | TFLOPs: 35.45 |
7:  iteration    11030/   11269 | consumed samples:      2823680 | consumed tokens:   5782896640 | elapsed time per iteration (s): 0.47 | learning rate: 2.020E-05 | global batch size:   256 | lm loss: 3.291371E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.203 | TFLOPs: 35.39 |
7:  iteration    11040/   11269 | consumed samples:      2826240 | consumed tokens:   5788139520 | elapsed time per iteration (s): 0.47 | learning rate: 2.019E-05 | global batch size:   256 | lm loss: 3.295230E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.232 | TFLOPs: 35.40 |
7:  iteration    11050/   11269 | consumed samples:      2828800 | consumed tokens:   5793382400 | elapsed time per iteration (s): 0.47 | learning rate: 2.017E-05 | global batch size:   256 | lm loss: 3.276016E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.159 | TFLOPs: 35.39 |
7:  iteration    11060/   11269 | consumed samples:      2831360 | consumed tokens:   5798625280 | elapsed time per iteration (s): 0.47 | learning rate: 2.016E-05 | global batch size:   256 | lm loss: 3.300319E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 539.176 | TFLOPs: 35.26 |
7:  iteration    11070/   11269 | consumed samples:      2833920 | consumed tokens:   5803868160 | elapsed time per iteration (s): 0.47 | learning rate: 2.014E-05 | global batch size:   256 | lm loss: 3.303376E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.022 | TFLOPs: 35.38 |
7:  iteration    11080/   11269 | consumed samples:      2836480 | consumed tokens:   5809111040 | elapsed time per iteration (s): 0.47 | learning rate: 2.013E-05 | global batch size:   256 | lm loss: 3.290928E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.267 | TFLOPs: 35.40 |
7:  iteration    11090/   11269 | consumed samples:      2839040 | consumed tokens:   5814353920 | elapsed time per iteration (s): 0.47 | learning rate: 2.011E-05 | global batch size:   256 | lm loss: 3.285753E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 541.041 | TFLOPs: 35.38 |
7:  iteration    11100/   11269 | consumed samples:      2841600 | consumed tokens:   5819596800 | elapsed time per iteration (s): 0.47 | learning rate: 2.010E-05 | global batch size:   256 | lm loss: 3.289551E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.470 | TFLOPs: 35.35 |
7:  iteration    11110/   11269 | consumed samples:      2844160 | consumed tokens:   5824839680 | elapsed time per iteration (s): 0.47 | learning rate: 2.009E-05 | global batch size:   256 | lm loss: 3.287623E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.974 | TFLOPs: 35.38 |
7:  iteration    11120/   11269 | consumed samples:      2846720 | consumed tokens:   5830082560 | elapsed time per iteration (s): 0.47 | learning rate: 2.008E-05 | global batch size:   256 | lm loss: 3.320606E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.801 | TFLOPs: 35.37 |
7:  iteration    11130/   11269 | consumed samples:      2849280 | consumed tokens:   5835325440 | elapsed time per iteration (s): 0.47 | learning rate: 2.007E-05 | global batch size:   256 | lm loss: 3.293619E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.704 | TFLOPs: 35.36 |
7:  iteration    11140/   11269 | consumed samples:      2851840 | consumed tokens:   5840568320 | elapsed time per iteration (s): 0.47 | learning rate: 2.006E-05 | global batch size:   256 | lm loss: 3.289527E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.700 | TFLOPs: 35.36 |
7:  iteration    11150/   11269 | consumed samples:      2854400 | consumed tokens:   5845811200 | elapsed time per iteration (s): 0.48 | learning rate: 2.005E-05 | global batch size:   256 | lm loss: 3.280003E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 538.265 | TFLOPs: 35.20 |
7:  iteration    11160/   11269 | consumed samples:      2856960 | consumed tokens:   5851054080 | elapsed time per iteration (s): 0.47 | learning rate: 2.004E-05 | global batch size:   256 | lm loss: 3.285721E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.821 | TFLOPs: 35.37 |
7:  iteration    11170/   11269 | consumed samples:      2859520 | consumed tokens:   5856296960 | elapsed time per iteration (s): 0.47 | learning rate: 2.004E-05 | global batch size:   256 | lm loss: 3.283740E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.779 | TFLOPs: 35.37 |
7:  iteration    11180/   11269 | consumed samples:      2862080 | consumed tokens:   5861539840 | elapsed time per iteration (s): 0.47 | learning rate: 2.003E-05 | global batch size:   256 | lm loss: 3.289019E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.591 | TFLOPs: 35.35 |
7:  iteration    11190/   11269 | consumed samples:      2864640 | consumed tokens:   5866782720 | elapsed time per iteration (s): 0.47 | learning rate: 2.002E-05 | global batch size:   256 | lm loss: 3.292329E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.760 | TFLOPs: 35.37 |
7:  iteration    11200/   11269 | consumed samples:      2867200 | consumed tokens:   5872025600 | elapsed time per iteration (s): 0.47 | learning rate: 2.002E-05 | global batch size:   256 | lm loss: 3.297464E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.514 | TFLOPs: 35.35 |
7:  iteration    11210/   11269 | consumed samples:      2869760 | consumed tokens:   5877268480 | elapsed time per iteration (s): 0.47 | learning rate: 2.001E-05 | global batch size:   256 | lm loss: 3.283976E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.638 | TFLOPs: 35.36 |
7:  iteration    11220/   11269 | consumed samples:      2872320 | consumed tokens:   5882511360 | elapsed time per iteration (s): 0.47 | learning rate: 2.001E-05 | global batch size:   256 | lm loss: 3.293830E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.567 | TFLOPs: 35.35 |
7:  iteration    11230/   11269 | consumed samples:      2874880 | consumed tokens:   5887754240 | elapsed time per iteration (s): 0.47 | learning rate: 2.001E-05 | global batch size:   256 | lm loss: 3.297632E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.161 | TFLOPs: 35.33 |
7:  iteration    11240/   11269 | consumed samples:      2877440 | consumed tokens:   5892997120 | elapsed time per iteration (s): 0.47 | learning rate: 2.000E-05 | global batch size:   256 | lm loss: 3.290126E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.695 | TFLOPs: 35.36 |
7:  iteration    11250/   11269 | consumed samples:      2880000 | consumed tokens:   5898240000 | elapsed time per iteration (s): 0.47 | learning rate: 2.000E-05 | global batch size:   256 | lm loss: 3.289125E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.887 | TFLOPs: 35.37 |
7:  iteration    11260/   11269 | consumed samples:      2882560 | consumed tokens:   5903482880 | elapsed time per iteration (s): 0.47 | learning rate: 2.000E-05 | global batch size:   256 | lm loss: 3.292947E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 540.668 | TFLOPs: 35.36 |
0: [after training is done] datetime: 2023-03-15 23:29:15 
0: saving checkpoint at iteration   11269 to checkpoints_280m5b9400m
7: -----------------------------------------------------------------------------------------------------------------
7: validation loss at the end of training for val data | lm loss value: 3.325924E+00 | lm loss PPL: 2.782471E+01 | 
7: -----------------------------------------------------------------------------------------------------------------
0: [2023-03-15 23:29:15,246] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step11269 is begin to save!
0: [2023-03-15 23:29:15,249] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_01-model_00-model_states.pt...
0: [2023-03-15 23:29:15,363] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_01-model_00-model_states.pt.
0: [2023-03-15 23:29:15,363] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_03-model_00-model_states.pt...
0: [2023-03-15 23:29:15,388] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_03-model_00-model_states.pt.
0: [2023-03-15 23:29:15,389] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_04-model_00-model_states.pt...
0: [2023-03-15 23:29:15,413] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_04-model_00-model_states.pt.
0: [2023-03-15 23:29:15,413] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_05-model_00-model_states.pt...
0: [2023-03-15 23:29:15,437] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_05-model_00-model_states.pt.
0: [2023-03-15 23:29:15,437] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_06-model_00-model_states.pt...
0: [2023-03-15 23:29:15,461] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_06-model_00-model_states.pt.
0: [2023-03-15 23:29:15,462] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_07-model_00-model_states.pt...
0: [2023-03-15 23:29:15,486] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_07-model_00-model_states.pt.
0: [2023-03-15 23:29:15,486] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_08-model_00-model_states.pt...
0: [2023-03-15 23:29:15,510] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_08-model_00-model_states.pt.
0: [2023-03-15 23:29:15,510] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_09-model_00-model_states.pt...
0: [2023-03-15 23:29:15,534] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_09-model_00-model_states.pt.
0: [2023-03-15 23:29:15,534] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_10-model_00-model_states.pt...
0: [2023-03-15 23:29:15,558] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_10-model_00-model_states.pt.
0: [2023-03-15 23:29:15,558] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_11-model_00-model_states.pt...
0: [2023-03-15 23:29:15,582] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_11-model_00-model_states.pt.
0: [2023-03-15 23:29:15,582] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_12-model_00-model_states.pt...
0: [2023-03-15 23:29:15,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_12-model_00-model_states.pt.
0: [2023-03-15 23:29:15,607] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_13-model_00-model_states.pt...
0: [2023-03-15 23:29:15,630] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_13-model_00-model_states.pt.
0: [2023-03-15 23:29:15,631] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_14-model_00-model_states.pt...
0: [2023-03-15 23:29:15,654] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_14-model_00-model_states.pt.
0: [2023-03-15 23:29:15,655] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_15-model_00-model_states.pt...
0: [2023-03-15 23:29:15,678] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_15-model_00-model_states.pt.
0: [2023-03-15 23:29:15,679] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_16-model_00-model_states.pt...
0: [2023-03-15 23:29:15,703] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_16-model_00-model_states.pt.
0: [2023-03-15 23:29:15,703] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_17-model_00-model_states.pt...
0: [2023-03-15 23:29:15,727] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_17-model_00-model_states.pt.
0: [2023-03-15 23:29:15,727] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_18-model_00-model_states.pt...
0: [2023-03-15 23:29:15,751] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_18-model_00-model_states.pt.
0: [2023-03-15 23:29:15,751] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_19-model_00-model_states.pt...
0: [2023-03-15 23:29:15,775] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_19-model_00-model_states.pt.
0: [2023-03-15 23:29:15,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_20-model_00-model_states.pt...
0: [2023-03-15 23:29:15,799] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_20-model_00-model_states.pt.
0: [2023-03-15 23:29:15,799] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/layer_22-model_00-model_states.pt...
0: [2023-03-15 23:29:15,800] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/layer_22-model_00-model_states.pt.
0: [2023-03-15 23:29:15,801] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_280m5b9400m/global_step11269/mp_rank_00_model_states.pt
0: [2023-03-15 23:29:15,801] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/mp_rank_00_model_states.pt...
0: [2023-03-15 23:29:15,803] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/mp_rank_00_model_states.pt.
0: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
0: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
2: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:29:15,820] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
7: [2023-03-15 23:29:15,821] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
1: [2023-03-15 23:29:15,879] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:29:15,879] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-15 23:29:15,879] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
0: [2023-03-15 23:29:15,880] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:29:15,880] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-15 23:29:15,880] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
5: [2023-03-15 23:29:15,881] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:29:15,881] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-15 23:29:15,881] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
7: [2023-03-15 23:29:15,881] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:29:15,882] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-15 23:29:15,882] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
0: [2023-03-15 23:29:15,882] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:29:15,882] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-15 23:29:15,882] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
0: [2023-03-15 23:29:15,882] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:29:15,882] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-15 23:29:15,882] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
0: [2023-03-15 23:29:15,883] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:29:15,884] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-15 23:29:15,884] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
5: [2023-03-15 23:29:15,885] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:29:15,887] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:29:15,887] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:29:15,887] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
0: [2023-03-15 23:29:15,887] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-15 23:29:15,887] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
7: [2023-03-15 23:29:15,887] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:29:15,887] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-15 23:29:15,887] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
2: [2023-03-15 23:29:15,887] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
1: [2023-03-15 23:29:15,888] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:29:15,888] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:29:15,888] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:29:15,888] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-15 23:29:15,888] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
7: [2023-03-15 23:29:15,888] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:29:15,888] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
7: [2023-03-15 23:29:15,888] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
1: [2023-03-15 23:29:15,888] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
7: [2023-03-15 23:29:15,888] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-15 23:29:15,888] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
7: [2023-03-15 23:29:15,888] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
4: [2023-03-15 23:29:15,889] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:29:15,889] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-15 23:29:15,889] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
4: [2023-03-15 23:29:15,889] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:29:15,889] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-15 23:29:15,889] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
3: [2023-03-15 23:29:15,889] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:29:15,889] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:29:15,889] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:29:15,890] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:29:15,890] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-15 23:29:15,890] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
7: [2023-03-15 23:29:15,890] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:29:15,890] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:29:15,890] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-15 23:29:15,890] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-15 23:29:15,891] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
7: [2023-03-15 23:29:15,891] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
0: [2023-03-15 23:29:15,891] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:29:15,891] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-15 23:29:15,891] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
2: [2023-03-15 23:29:15,892] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:29:15,892] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:29:15,892] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:29:15,892] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-15 23:29:15,892] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-15 23:29:15,892] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
2: [2023-03-15 23:29:15,892] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-15 23:29:15,892] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
2: [2023-03-15 23:29:15,892] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
4: [2023-03-15 23:29:15,892] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:29:15,892] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-15 23:29:15,892] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
4: [2023-03-15 23:29:15,892] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:29:15,893] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-15 23:29:15,892] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:29:15,892] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:29:15,893] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
4: [2023-03-15 23:29:15,893] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-15 23:29:15,893] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-15 23:29:15,893] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
4: [2023-03-15 23:29:15,893] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
4: [2023-03-15 23:29:15,892] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:29:15,893] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-15 23:29:15,893] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
4: [2023-03-15 23:29:15,895] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-15 23:29:15,895] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-15 23:29:15,895] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
7: [2023-03-15 23:29:15,897] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:29:15,897] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-15 23:29:15,898] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
7: [2023-03-15 23:29:15,898] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-15 23:29:15,898] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-15 23:29:15,898] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
2: [2023-03-15 23:29:15,899] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:29:15,899] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-15 23:29:15,899] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
2: [2023-03-15 23:29:15,899] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:29:15,899] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-15 23:29:15,899] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-15 23:29:15,899] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-15 23:29:15,899] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
2: [2023-03-15 23:29:15,899] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
6: [2023-03-15 23:29:15,901] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:29:15,901] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:29:15,901] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-15 23:29:15,901] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-15 23:29:15,901] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
6: [2023-03-15 23:29:15,901] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
6: [2023-03-15 23:29:15,901] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:29:15,901] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:29:15,901] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-15 23:29:15,901] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-15 23:29:15,901] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
6: [2023-03-15 23:29:15,901] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
6: [2023-03-15 23:29:15,902] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:29:15,902] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-15 23:29:15,902] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
0: [2023-03-15 23:29:15,902] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:29:15,902] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
0: [2023-03-15 23:29:15,902] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:29:15,902] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-15 23:29:15,902] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:29:15,902] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
1: [2023-03-15 23:29:15,902] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
0: [2023-03-15 23:29:15,903] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-15 23:29:15,903] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
1: [2023-03-15 23:29:15,903] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
1: [2023-03-15 23:29:15,903] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:29:15,903] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-15 23:29:15,903] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
1: [2023-03-15 23:29:15,903] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:29:15,903] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-15 23:29:15,903] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
3: [2023-03-15 23:29:15,889] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:29:15,889] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:29:15,889] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-15 23:29:15,889] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-15 23:29:15,889] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-15 23:29:15,889] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-15 23:29:15,889] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-15 23:29:15,889] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
3: [2023-03-15 23:29:15,889] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
3: [2023-03-15 23:29:15,889] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
3: [2023-03-15 23:29:15,889] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
3: [2023-03-15 23:29:15,890] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
3: [2023-03-15 23:29:15,899] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:29:15,899] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-15 23:29:15,899] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
3: [2023-03-15 23:29:15,899] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:29:15,899] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-15 23:29:15,899] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
3: [2023-03-15 23:29:15,903] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-15 23:29:15,903] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-15 23:29:15,903] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
6: [2023-03-15 23:29:15,906] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:29:15,906] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-15 23:29:15,906] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
6: [2023-03-15 23:29:15,907] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:29:15,907] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-15 23:29:15,907] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-15 23:29:15,907] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-15 23:29:15,907] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
6: [2023-03-15 23:29:15,907] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
5: [2023-03-15 23:29:15,885] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-15 23:29:15,885] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
5: [2023-03-15 23:29:15,887] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:29:15,887] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-15 23:29:15,887] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
5: [2023-03-15 23:29:15,887] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:29:15,887] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-15 23:29:15,887] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
5: [2023-03-15 23:29:15,890] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:29:15,890] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-15 23:29:15,890] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
5: [2023-03-15 23:29:15,891] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:29:15,891] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-15 23:29:15,891] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
5: [2023-03-15 23:29:15,891] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:29:15,891] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-15 23:29:15,891] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
5: [2023-03-15 23:29:15,901] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-15 23:29:15,901] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-15 23:29:15,901] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
1: [2023-03-15 23:29:15,929] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-15 23:29:15,929] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-15 23:29:15,929] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
0: [2023-03-15 23:29:15,967] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_280m5b9400m/global_step11269/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-15 23:29:15,967] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step11269 is ready now!
0:   successfully saved checkpoint at iteration   11269 to checkpoints_280m5b9400m
END 3318400: Wed 15 Mar 2023 11:29:24 PM EET